zfcpdump kernel can not be IPLed, wrong file name

[bugproxy]

[Impact]

 * zfcpdump-kernel incompatible with s390-tools in focal, due to wrong file name, needed in focal and up
 * zfcpdump-kernel incompatible with SIPL (secure IPL), and will not be fixed.

 * Upgrade to the v5.4 kernel
 * Signing will not be enabled
 * Update the path of the image, to the one that `zipl -d` in focal expects

[Test Case]

 * Prepare sda1 drive for SCSI dump
 * Stop all processes from the HMC
 * Perform SCSI dump load from HMC
 * Observe that dump is successful and used v5.4 kernel from the Operating System Messages
 * Perform regular boot
 * Mount the dump, and observe it is there in full

 * Can be performed on the canonical z13 hmc without SIPL
 * zfcpdump with secureboot will not be possible

[Publication]

 * zfcpdump-kernel image is OS series independant, and thus can be build in focal with copies up to groovy.

[Regression Potential]

 * The kernel image used for zfcpdump is fairly static, doesn't have loadable modules, but it does allow reading kernel memory which in theory is not in the same spirit as lockdown. However, stopping all processes and triggering scsi-dump is a priviledged HMC operation that is otheriwse has a much higher access restrictions than lockdown can provide.

[Other Info]

 * Original bug report

I installed Ubuntu 20.04 on IBM z15 with secure=1 in zipl conf.
System can be secure booted, /sys/firmware/ipl/secure shows "1".
I prepared zfcp dump disk as described in LTC bug 185713.
Stopped the system and performed a SCSI dump with "Enable Secure Boot for Linux" enabled.

Operating System Messages on HMC:
Preparing system.
Starting system.
System version 8.
Watchdog enabled.
Running 'ZBootLoader' version '1.0.0' level 'D41C.D41C_0014'.
ZBootLoader 2.1.0.
MLOLOA6269064E Secure IPL: There are no signed components available on device HB
A=0.0.1800, WWPN=500507630309D327, LUN=4046400900000000.
IPL failed.

Without "Enable Secure Boot for Linux" the dump kernel was IPLed and a dump created.

Then I tried to rewrite the zfcp dump kernel with explicite setting of --secure=1:
root@t35lp25:~# zipl --secure=1 -d /dev/disk/by-id/scsi-36005076303ffd3270000000000004609-part1
Building bootmap directly on partition '/dev/disk/by-id/scsi-36005076303ffd3270000000000004609-part1'
Adding dump section
  initial ramdisk...: /lib/s390-tools/zfcpdump/zfcpdump-initrd
  kernel image......: /lib/s390-tools/zfcpdump/zfcpdump-image
  kernel parmline...: 'root=/dev/ram0 dump_mem=1 possible_cpus=1 cgroup_disable=memory '
  component address:
    heap area.......: 0x00002000-0x00005fff
    stack area......: 0x0000f000-0x0000ffff
    internal loader.: 0x0000a000-0x0000dfff
    parameters......: 0x00009000-0x000091ff
    kernel image....: 0x00010000-0x001b9fff
    parmline........: 0x001ba000-0x001ba1ff
    initial ramdisk.: 0x001c0000-0x0020edff
Preparing boot device: sde.
Done.

...and tried to SCSI dump this device again. But the same failure occured.
Again, without "Enable Secure Boot for Linux" the dump kernel was IPLed and a dump created.

[Dimitri John Ledkov (xnox)]

We can either revert the path change in s390-tools or rebuild the zfcpdump kernel flavour with the new name.

[Dimitri John Ledkov (xnox)]

(this is due to s390-tools renaming the expected zfcpdump kernel name)

Separately, are you expecting for the zfcpdump-kernel to be secure boot signed? because currently it is not.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-05-06 08:28 EDT-------
(In reply to comment #6)
> We can either revert the path change in s390-tools or rebuild the zfcpdump
> kernel flavour with the new name.

This should IMO be decided by the s390tools maintainer! (I personally prefer the latter.)

------- Comment From <email address hidden> 2020-05-06 08:34 EDT-------
(In reply to comment #7)
> Separately, are you expecting for the zfcpdump-kernel to be secure boot
> signed? because currently it is not.

Yes: If a user has booted his system with Secure Boot enables and needs to perform a standalone zfcp dump, the HMC panel keeps the setting of Secure Boot enablement. Thus, if it was enabled for normal zfcp boot, it remains enabled for zfcp dump, and if the zfcpdump kernel is not signed, the dump fails and probably part of the memory content is lost. Secondly, if dumpconf is being used, the automatic zfcpdump will fail, too.

[Dimitri John Ledkov (xnox)]

In that case we will need to create zfcpdump-signed-kernel & backport that to focal.

[Dimitri John Ledkov (xnox)]

Hm, I'm not sure we can sign the zfcpdump-kernel.

By convention, in Focal, signed kernels enforce signed module loading & lockdown that prevents unsigned module loading, kexec unsigned kernels or reading arbitrary kernel memory from userspace. And I am under impression that zfcpdump kernel/initrd rely on being able to read kernel memory.

The zfcpdump-kernel flavour currently is built using zfcpdump_defconfig. I would be more comfortable if we could use the stock signed kernel image as the zfcpdump one, instead of the purpose built one. And include any missing modules in the zfcpdump initrd and/or adjust the cmdline to do things like PANIC_ON_OOPS=y. But i guess we will not get CONFIG_CC_OPTIMIZE_FOR_SIZE=y with the stock kernel image.

Does zfcdump work with locked-down kernels?
Why do we want/prefer a separate kernel flavour for zfcpdump, instead of the stock one?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-05-12 11:56 EDT-------
Hi xnox,

we need a separate flavor because zfcpdump kernels (on pre z15) are limited to 64M of memory. Any stock kernel would run oom in such a constrained environment.

(In reply to comment #11)
> Hm, I'm not sure we can sign the zfcpdump-kernel.
>
> By convention, in Focal, signed kernels enforce signed module loading &
> lockdown that prevents unsigned module loading, kexec unsigned kernels or
> reading arbitrary kernel memory from userspace. And I am under impression
> that zfcpdump kernel/initrd rely on being able to read kernel memory.

hmm... not sure if this is really a problem
* the kernel is build without CONFIG_MODULES so all of the non existing kernel modules are automatically signed
* not sure how lockdown works in detail but zfcpdump only needs access to /proc/vmcore which should still be possible. Otherwise kdump would be broken as well.
* kexec is not needed. init is replaced by a piece of code that simply copies /proc/vmcore to disc and reboots.

> The zfcpdump-kernel flavour currently is built using zfcpdump_defconfig. I
> would be more comfortable if we could use the stock signed kernel image as
> the zfcpdump one, instead of the purpose built one. And include any missing
> modules in the zfcpdump initrd and/or adjust the cmdline to do things like
> PANIC_ON_OOPS=y. But i guess we will not get CONFIG_CC_OPTIMIZE_FOR_SIZE=y
> with the stock kernel image.

As said before the stock kernel won't run on a pre z15 machine. On z15 (needed for secure boot anyway) that might be an option as the HSA area (the piece of memory the zfcpdump kernel runs in) was increased to 512M. That's more than usually reserved for a kdump kernel, and should be enough for any stock kernel.

The problem is that zfcpdumps design never considered such an option so it won't run out of the box. Furthermore you would then need to support two different dump methods depending on the machine generation you are running on as long as any pre z15 machine is supported.

I must admit that getting rid of this monstrosity would be great but I don't think it will happen any time soon.

> Does zfcdump work with locked-down kernels?

Not sure never tried.

Philipp

[Dimitri John Ledkov (xnox)]

Thank you for those details.

For example, if and when we bump ubuntu minimum abi to z15 we might be able to kill the zfcpdump kernel.

If there are no modules i guess any initrd will not be able to do much.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-05-20 07:39 EDT-------
(In reply to comment #14)
> Thank you for those details.
>
> For example, if and when we bump ubuntu minimum abi to z15 we might be able
> to kill the zfcpdump kernel.

That's at least my hope. There's an other problem I keep forgetting. Currently only LPAR has the bigger HSA size. z/VM has to emulate the HSA and cannot cope with the bigger size yet. Not sure if/how KVM handles this. So simply bumping the ALS won't be enough. You still need to keep an eye on the hypervisors...

> If there are no modules i guess any initrd will not be able to do much.

Actually my hope is we find a way to get it work with the standad kernel + kdump initrd. So basically getting a firmware assisted kdump.

------- Comment From <email address hidden> 2020-05-20 07:41 EDT-------
I also played around a little bit more with zfcpdump + secure boot and I'm no longer sure that simply signing the zfcpdump kernel is enough. There might also be a bug in zipl. But that needs further investigation.

[Frank Heimes (fheimes)]

It's in the new queue since 2020-07-08:
https://launchpad.net/ubuntu/focal/+queue?queue_state=0&queue_text=zfcpdump-kernel

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-07-31 07:14 EDT-------
This is currently not supported.
As of today there is no code in zipl that writes signature entries for a dump kernel.
This was not part of the initial item.

It might be addressed in the future.

[Dimitri John Ledkov (xnox)]

In that case we should not be signing zfcpdump-kernel for now.

To minimize amount of things the sipl key signs.

[Frank Heimes (fheimes)]

@IBM Can we agree to close this then? (based on comments 8, 10 and 11)

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-08-26 09:14 EDT-------
IBM Bugzilla status-> closed

[Robie Basak (racb)]

SRU review: as discussed on IRC this needs further documentation please:

Why is an update to the 5.4 kernel required and how can we ensure that existing users on Focal are not regressed?

Why is this not being fixed in Groovy first, as is the normal SRU policy?

Thanks

[Launchpad Janitor (janitor)]

This bug was fixed in the package zfcpdump-kernel - 5.4-0ubuntu2

---------------
zfcpdump-kernel (5.4-0ubuntu2) groovy; urgency=medium

  * Switch to 5.4 base from focal (5.4.0-49.53).
  * Drop fix to zfcpdump config, included in 5.4.
  * Rename zfcpdump_part.image, to zfcpdump-image, to match current
    s390-tools expectations. LP: #1877089

 -- Dimitri John Ledkov <email address hidden> Mon, 06 Jul 2020 13:05:49 +0100

[Brian Murray (brian-murray)]

Could somebody address Robie's remaining questions in comment #15?

[Frank Heimes (fheimes)]

@brian-murray and @acb
This bug is in between 'Fix Released' for groovy.
Please see also zfcpdump-kernel changelog:
zfcpdump-kernel (5.4-0ubuntu2) groovy; urgency=medium
  * Switch to 5.4 base from focal (5.4.0-49.53).
  * Drop fix to zfcpdump config, included in 5.4.
  * Rename zfcpdump_part.image, to zfcpdump-image, to match current
    s390-tools expectations. LP: #1877089
The zfcpdump kernel got now updated to the focal kernel.

I think that this bug was raised right after focal was released (and probably before groovy development was really open) - it had initially only a series entry for focal and it was at the beginning assumed that only focal is affected (groovy s390-tool was not there yet).
It took a while to get this topics discussed and achieve more clarification (and the bug states changed quite a bit).
At some point in time a first upload was done on focal - that got dropped then and replaced by a revised version that still sits in focal unapproved (since 2020-09-17)
https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=zfcpdump-kernel

Meanwhile groovy evolved and it became clear that this needs to be tacked too, but the first focal upload was already out. Hence the groovy upload came late, means after focal (so too late according to the SRU process, hence comment #15).

Since the zfcpdump-kernel package/image is quite self sufficient and static, the regression risk is very low, since the zfcpdump-kernel image is even independent from the OS series, hence it can be build in one series (now focal) and then be used in focal or newer. Please see also bug description: [Publication] and [Regression Potential]

[Dimitri John Ledkov (xnox)]

On Tue, Nov 3, 2020 at 10:15 PM Brian Murray <email address hidden>
wrote:

> Could somebody address Robie's remaining questions in comment #15?
>
>
Rovies' questions got address the same day.

Upload was made to groovy.

This kernel image has no relationship to the running OS, it is a kdump-like
kernel image, which firmware (HMC) loads whilst all processes and execution
is stopped. Thus is independent from the OS/kernel/userspace that is being
dumped.

However, the filenames in the package must be compatible with the userspace
tooling to prepare firmware to load this kernel.

Jump to v5.4 is because 4.15 will go end of life and out of support, whilst
focal one will remain. Nonetheless it is not expected that there will be
further updates to the zfcpdump kernel, the config of this kernel flavour
is tiny and doesn't have network access thus is hard to exploit, and it is
not signed for secureboot.

--
Regards,

Dimitri.

[Brian Murray (brian-murray)]

Hello bugproxy, or anyone else affected,

Accepted zfcpdump-kernel into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfcpdump-kernel/5.4-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-12-10 13:43 EDT-------
I updated to https://launchpad.net/ubuntu/focal/+queue?queue_state=3&queue_text=zfcpdump-kernel as of focal-proposed.
Then I zipled the SCSI dump LUN:
root@t35lp25:~# zipl -d /dev/disk/by-id/scsi-36005076303ffd3270000000000004609-part1
Building bootmap directly on partition '/dev/disk/by-id/scsi-36005076303ffd3270000000000004609-part1'
Adding dump section
kernel image......: /lib/s390-tools/zfcpdump/zfcpdump-image
kernel parmline...: 'root=/dev/ram0 dump_mem=1 possible_cpus=1 cgroup_disable=memory '
component address:
heap area.......: 0x00002000-0x00005fff
stack area......: 0x0000f000-0x0000ffff
internal loader.: 0x0000a000-0x0000dfff
parameters......: 0x00009000-0x000091ff
kernel image....: 0x00010000-0x001b7fff
parmline........: 0x001b8000-0x001b81ff
Preparing boot device: sdi.
Done.
... and dumped from LUN with secure boot enabled.

But it still shows the same symptom:
...
Running 'ZBootLoader' version '2.1.4' level 'D41C.D41C_0020'.
MLOLOA6269064E Secure IPL: There are no signed components available on device HB
A=0.0.1800, WWPN=500507630309D327, LUN=4046400900000000.
IPL failed (110).

I zipled again with additional switch --secure=1, but that didn't help either.

Please change the tag to verification-failed-focal.

[Dimitri John Ledkov (xnox)]

@thorsten

that is invalid validation.

No IBM Z mainframe currently supports zfcpdump with secureboot on.

You must validate this with secureboot turned off. Only then HMC supports to zfcpdump.

The issue we are fixing in this update that `zipl -d` could not find zfcpdump-kernel image since in the release package it was named under the old name of /lib/s390-tools/zfcpdump/zfcpdump_part.image instead of the required name of /lib/s390-tools/zfcpdump/zfcpdump-image.

Please confirm that zfcpdump is successful with the zfpcdump-kernel from proposed & secureboot turned off in the HMC.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-12-11 03:37 EDT-------
Hi Dimitri, now I understood.
As mentioned in LP1876715 (LTC 185713) in my last recent comments, the naming problem is fixed with zfcpdump-kernel 5.4-0ubuntu1.

[Frank Heimes (fheimes)]

Ok, in this case I'll set the tag to done - thx Thorsten

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for zfcpdump-kernel has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package zfcpdump-kernel - 5.4-0ubuntu1

---------------
zfcpdump-kernel (5.4-0ubuntu1) focal; urgency=medium

  * Switch to 5.4 base from focal.
  * Drop fix to zfcpdump config, included in 5.4.
  * Rename zfcpdump_part.image, to zfcpdump-image, to match current
    s390-tools expectations. LP: #1877089

 -- Dimitri John Ledkov <email address hidden> Mon, 06 Jul 2020 13:05:49 +0100