[23.10] Please test secure-boot and lockdown on the early 6.5 kernel (s390x)

[Andrea Righi]

The Canonical kernel team is working on a kernel 6.5 for 'mantic' (23.10) and has some first kernel builds ready for secure-boot and lockdown testing (version 6.5.0-1.1).

To avoid potential negative implications that a broken secure-boot lockdown functionality, shipped with a new major kernel version, would cause (esp. using the production key), we ask as usual to get secure-boot lockdown tested (with every new major kernel release) early in the cycle, based on the following test kernel in the kernel team's PPA, using a test key for the signature.

The early test build is available in: ppa:canonical-kernel-team/unstable
(https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/unstable/)

The PPA key used for signing can be found in the tarball available here:
https://ppa.launchpad.net/canonical-kernel-team/unstable/ubuntu/dists/devel/main/signed/linux-generate-unstable-s390x/current/

(Please note that this kernel is coming from the 'canonical-kernel-team' PPA, hence it is NOT signed with the regular archive/release/production key, BUT instead with the above PPA test signing key!)

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-07-27 05:46 EDT-------
we installed from ppa:canonical-kernel-team/unstable:

cat /etc/os-release
PRETTY_NAME="Ubuntu Mantic Minotaur (development branch)"
NAME="Ubuntu"
VERSION_ID="23.10"
VERSION="23.10 (Mantic Minotaur)"
VERSION_CODENAME=mantic
...

# uname -r
6.5.0-1-generic

and used 6.5.0-1.1+1/control/sipl.x509 signature and 6.5.0-1.1+1/boot/vmlinuz-6.5.0-1-generic.sipl from the tar file from https://ppa.launchpad.net/canonical-kernel-team/unstable/ubuntu/dists/devel/main/signed/linux-generate-unstable-s390x/current/
signed.tar.gz

ls -l /boot/vmlinuz
lrwxrwxrwx 1 root root 28 Jul 26 11:32 /boot/vmlinuz -> vmlinuz-6.5.0-1-generic.sipl

load with kernel vmlinuz-6.5.0-1-generic.sipl
- without secure boot enable
- without adding the signature
==> as expected:

IPB received.
IPB sent.
System version 9.
Watchdog enabled.
Running 'ZBootLoader' version '3.2.2' level 'D51C.D51C_328.16'.
OK00000000 Success

load with kernel vmlinuz-6.5.0-1-generic
- with secure boot enable
- with adding the signature
==> not expected:
IPB received.
IPB sent.
System version 9.
Watchdog enabled.
Running 'ZBootLoader' version '3.2.2' level 'D51C.D51C_328.16'.
--- Audit message summary start ---
MLOLOA62693210 Audit: Signature verification failure for component 3 in program
0 loaded from device HBA:0.0.1900,WWPN:500507630710572C,LUN:4020404E00000000.
--- Audit message summary end ---
MLOLOA6269321F A security violation error was encountered when loading from devi
ce HBA:0.0.1900,WWPN:500507630710572C,LUN:4020404E00000000.
IPL failed (110).

LdiplStructureProcessor.cpp:processComponentTable:75: Found IMAGE component @0x156d1630 w LOAD ADDRESS 0xa000.
CertificateVerification.cpp:verifySignature:237: Failed to verify image component @0x156d1630 w associated signature component @0x156d19b0 and certificate #0 w vc_index 1.
LdiplStructureProcessor.cpp:checkSignature:288: Audit: Signature verification failed for component #3 @0x156d1630.
LdiplStructureProcessor.cpp:processComponentTable:91: Found Image Component @0x156d1630 w SCLAB @0x156cde10.
LdiplStructureProcessor.cpp:processComponentTable:98: Found the 'global' SCLAB (1) @0x156d1630
LdiplComponentType02Image.cpp:checkSignedCodeLoadingAttributesFacilityRules:235: Audit summary 2a: Error indicators for the SCLAB of component 3 is 0x0.

but load for component 5 worked:
LdiplStructureProcessor.cpp:processComponentTable:75: Found IMAGE component @0x156fcd90 w LOAD ADDRESS 0x10000.
CertificateVerification.cpp:verifySignature:227: Successfully verified image component @0x156fcd90 w associated signature component @0x156fef00 and certificate #0 w vc_index 1.
LdiplStructureProcessor.cpp:processComponentTable:91: Found Image Component @0x156fcd90 w SCLAB @0x156fcd50.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-07-27 09:45 EDT-------
Thanks Thomas for testing.

@Frank / Canonical:
Due to Thomas' test results ("signature verification failed") we suspect that there is a mismatch of the provided kernel and the signing key.
Can you please double check and maybe provide a new PPA kernel and / or corresponding key? Thanks

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-08-04 07:05 EDT-------
Setting status to "REOPENED" as we are waiting for new PPA kernel and / or corresponding key from Canonical for another test run

[Andrea Righi (arighi)]

Could you repeat the test with the latest kernel linux 6.5.0-4.4, available in ppa:canonical-kernel-team/unstable (https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/unstable/).

The signing key that needs to be enrolled is the one from the unstable ppa, as usual:
https://ppa.launchpad.net/canonical-kernel-team/unstable/ubuntu/dists/devel/main/signed/linux-generate-unstable-s390x/6.5.0-4.4/signed.tar.gz

Thanks!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-08-17 03:32 EDT-------
we installed from ppa:canonical-kernel-team/unstable:
# cat /etc/os-release
PRETTY_NAME="Ubuntu Mantic Minotaur (development branch)"
NAME="Ubuntu"
VERSION_ID="23.10"
VERSION="23.10 (Mantic Minotaur)"
VERSION_CODENAME=mantic
...
...
# uname -r
6.5.0-4-generic
# grep [0-9] /sys/firmware/ipl/*sec*
/sys/firmware/ipl/has_secure:1
/sys/firmware/ipl/secure:0
# ls -l /boot/vmlinuz /boot/initrd.img
lrwxrwxrwx 1 root root 26 Aug 15 09:19 /boot/initrd.img -> initrd.img-6.5.0-4-generic
lrwxrwxrwx 1 root root 23 Aug 15 09:19 /boot/vmlinuz -> vmlinuz-6.5.0-4-generic

load with kernel vmlinuz-6.5.0-4-generic
- without secure boot enable
- without adding the signature

IPB received.
IPB sent.
System version 9.
Watchdog enabled.
Running 'ZBootLoader' version '3.2.2' level 'D51C.D51C_328.16'.
--- Audit message summary start ---
MLOLOA62693210 Audit: Signature verification failure for component 3 in program
0 loaded from device HBA:0.0.1900,WWPN:500507630710572C,LUN:4020404E00000000.
--- Audit message summary end ---
OK00000000 Success

load with kernel vmlinuz-6.5.0-4-generic
- with secure boot enable
- without adding the signature

IPB received.
IPB sent.
System version 9.
Watchdog enabled.
Running 'ZBootLoader' version '3.2.2' level 'D51C.D51C_328.16'.
--- Audit message summary start ---
MLOLOA62693210 Audit: Signature verification failure for component 3 in program
0 loaded from device HBA:0.0.1900,WWPN:500507630710572C,LUN:4020404E00000000.
--- Audit message summary end ---
MLOLOA6269321F A security violation error was encountered when loading from devi
ce HBA:0.0.1900,WWPN:500507630710572C,LUN:4020404E00000000.
IPL failed (110).

load with kernel vmlinuz-6.5.0-4-generic
- with secure boot enable
- with adding the signature

IPB received.
IPB sent.
System version 9.
Watchdog enabled.
Running 'ZBootLoader' version '3.2.2' level 'D51C.D51C_328.16'.
--- Audit message summary start ---
MLOLOA62693210 Audit: Signature verification failure for component 3 in program
0 loaded from device HBA:0.0.1900,WWPN:500507630710572C,LUN:4020404E00000000.
--- Audit message summary end ---
MLOLOA6269321F A security violation error was encountered when loading from devi
ce HBA:0.0.1900,WWPN:500507630710572C,LUN:4020404E00000000.
IPL failed (110).

==> boot with secure boot enable and adding the certificate still didn't work

we used this Certificate:
openssl x509 -text -in sipl.x509
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
a1:b6:a0:75:09:df:f4:18
Signature Algorithm: sha512WithRSAEncryption
Issuer: CN = PPA canonical-kernel-team unstable SIPL
Validity
Not Before: Aug 23 20:47:25 2019 GMT
Not After : Aug 20 20:47:25 2029 GMT
Subject: CN = PPA canonical-kernel-team unstable SIPL
...
...

this was tested on our Z16 machine

[Andrea Righi (arighi)]

Alright, since I couldn't find any obvious changes in the signing process, I would assume that the regression has been introduced in the kernel itself at some point. The 6.2 kernel was good (and it was verified in the same ppa), the 6.5 kernel is failing, so let's try to do a sort of bisect, I've prepared a 6.4 kernel (6.4.0-2.2) in the same ppa:

 ppa:canonical-kernel-team/unstable (https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/unstable)

And the signing data:

 https://ppa.launchpad.net/canonical-kernel-team/unstable/ubuntu/dists/devel/main/signed/linux-generate-s390x/6.4.0-2.2/signed.tar.gz

It would be great if you could repeat the test also with this one. Thanks!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-08-22 07:22 EDT-------
we installed from ppa:canonical-kernel-team/unstable:
# cat /etc/os-release
PRETTY_NAME="Ubuntu Mantic Minotaur (development branch)"
NAME="Ubuntu"
VERSION_ID="23.10"
VERSION="23.10 (Mantic Minotaur)"
VERSION_CODENAME=mantic
...
...
# uname -r
6.5.0-4-generic
# grep [0-9] /sys/firmware/ipl/*sec*
/sys/firmware/ipl/has_secure:1
/sys/firmware/ipl/secure:0

# ls -l /boot/vmlinuz /boot/initrd.img
lrwxrwxrwx 1 root root 26 Aug 15 09:19 /boot/initrd.img -> initrd.img-6.5.0-4-generic
lrwxrwxrwx 1 root root 23 Aug 22 08:24 /boot/vmlinuz -> vmlinuz-6.5.0-4-generic

load with kernel vmlinuz-6.5.0-4-generic
- with secure boot enable
- with adding both PPA-keys
-> prod PPA key from 5.13.0-19.19 kernal packages
-> unstable PPA key from: https://ppa.launchpad.net/canonical-kernel-team/unstable/ubuntu/dists/devel/main/signed/linux-generate-unstable-s390x/6.5.0-4.4/signed.tar.gz

IPB received.
IPB sent.
System version 9.
Watchdog enabled.
Running 'ZBootLoader' version '3.2.2' level 'D51C.D51C_328.16'.
OK00000000 Success
[ 0.074381] Linux version 6.5.0-4-generic (buildd@bos01-s390x-015) (s390x-linux-gnu-gcc-13 (Ubuntu 13.2.0-1ubuntu1) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.40.90.20230720) #4-Ubuntu SMP Mon Jul 31 16:02:15 UTC 2023 (Ubuntu 6.5.0-4.4-generic 6.5.0-rc4)
[ 0.074382] setup: Linux is running natively in 64-bit mode
[ 0.074383] setup: Linux is running with Secure-IPL enabled
[ 0.074383] setup: The IPL report contains the following components:
[ 0.074384] setup: 0000000000002000 - 0000000000006000 (not signed)
[ 0.074385] setup: 0000000000009000 - 0000000000009200 (not signed)
[ 0.074386] setup: 000000000000a000 - 000000000000e000 (signed, verified)
[ 0.074387] setup: 000000000000f000 - 0000000000010000 (not signed)
[ 0.074388] setup: 0000000000010000 - 0000000000974000 (signed, verified)
[ 0.074389] setup: 0000000000975000 - 0000000000975200 (not signed)
[ 0.074390] setup: 0000000000985000 - 0000000000986000 (not signed)
[ 0.074391] setup: 0000000000990000 - 00000000028ada00 (not signed)
[ 0.074392] Kernel is locked down from Secure IPL mode; see man kernel_lockdown.7

after load:
# grep [0-9] /sys/firmware/ipl/*sec*
/sys/firmware/ipl/has_secure:1
/sys/firmware/ipl/secure:1

[Andrea Righi (arighi)]

I am a bit confused... According to comment #7 it looks like 6.5.0-4 passed the test, but the same kernel in comment #5 didn't pass the test (hence I uploaded 6.4.0-2 in the same ppa - comment #6).

Just want to make sure it's not a typo, if you confirm that 6.5.0-4 passed the secure boot test I will proceed to promote this kernel and sign it with the production key.

Thanks.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-08-29 08:32 EDT-------
@Andrea: Sorry for the confusion / the error on our side with the first round of testing.
We found out that the first test of kernel 6.5.0-4 (see LP comment #5) had failed because the system where the test ended up running didn't have the right / matching key loaded.
After double-checking that the test PPA key that you provided was uploaded successfully, the secure boot test with kernel 6.5.0-4 ran as expected (see LP comment #7).

With this, yes, we can confirm that 6.5.0-4 passed the secure boot test.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-oem-6.5/6.5.0-1004.4 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-oem-6.5' to 'verification-done-jammy-linux-oem-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-oem-6.5' to 'verification-failed-jammy-linux-oem-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!