[24.04] Power guest secure boot with static keys: kernel portion

[bugproxy]

== Comment: #2 - Daniel John Axtens <email address hidden> - 2020-11-05 20:15:10 ==
This is the kernel side of changes needed for LPAR/guest secure boot.

Because Ubuntu keeps its kernels so wonderfully up to date, I don't think there are any extra patches you need to pick up. (I'll double-check against the 21.04 tree once my git pulls finish!)

However, we potentially need some configuration changes to make sure kexec-ing into a crashdump kernel still works.

Because Lockdown requires that kexec kernels are signed by a key trusted by IMA, the public key for used for signing the kdump kernel needs to be in the IMA keyring or the platform keyring. For host secure boot (and in the UEFI case), it's loaded into the platform keyring. But in the case of guest secure boot with static keys, it's not loaded into the platform keyring so it needs to be loaded into the IMA keyring.

This is easy enough to do. Firstly, load the Secure Boot CA into the .primary_trusted_keys keyring via the CONFIG_SYSTEM_TRUSTED_KEYS property. We assume the key used to sign the kernel is signed by this CA.

Then, enable IMA_LOAD_X509, which allows certificates signed by a key on the .primary_trusted_keys keyring to be loaded into the IMA keyring. Then set IMA_X509_PATH to provide a path to the signing key on installed file system. (It may also be possible to do this step in userspace, so long as the CA is trusted by the kernel.)

Then that key will be loaded into the .ima keyring at boot and be used to appraise the kexec kernel for crashdumps.

[Frank Heimes (fheimes)]

Hi Daniel, btw. in which upstream kernel did the kernel patches for this landed (or will land)?
Just to be sure - 5.10 or 5.11 or already in with an earlier version?

[Dimitri John Ledkov (xnox)]

yes kernel config changes will be needed for this.

[Frank Heimes (fheimes)]

Just cross-referencing, this is the grub part: LP 1903289

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-11-12 01:12 EDT-------
Hi,

So, here are what I believe are the relevant kernel changes.

Firstly, there is a common core with the support for OpenPower/PowerNV Secure Boot - LP#1866909 and friends. This covers things like securing kexec under lockdown, and all went in for 20.04.

For this iteration of LPAR/KVM guest secure boot - which is supporting only static keys - we just need an additional patch to detect the slightly different Device Tree properties that are used to indicate guest vs host secure boot. This is commit 61f879d97ce4 ("powerpc/pseries: Detect secure and trusted boot state of the system.") which went into 5.9.

Kind regards,
Daniel

[Andrew Cloke (andrew-cloke)]

To confirm, this bug only requires that commit 61f879d97ce4 ("powerpc/pseries: Detect secure and trusted boot state of the system.") lands in hirsute. Is that correct, or are other patches also required?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-11-19 20:25 EDT-------
Hi,

I think that's the only feature patch required. There's not a lot because at this stage it's all based on static keys. So unlike the OpenPower secure boot, there's no code to interact with keys stored in firmware.

There is one config change that we also need: because there are no keys advertised by firmware or early boot, and because lockdown requires that kexec kernels be signed, we need to get the kernel signing key into the .ima keyring somehow. One way to do this is at boot with IMA_X509_PATH, but it can also be done at runtime. In either case, however, the CA that signs the kernel signing key needs to be built in to the kernel's .builtin_trusted_keys keyring. I haven't attempted this because I don't know much about how the signing process works in your build infrastructure, but I'm happy to help.

Of course, I expect there will also be bug fixes later!

Kind regards,
Daniel

[Frank Heimes (fheimes)]

I had a look at our 'Ubuntu unstable' 5.10 tree:
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/unstable
(that will once become the hirsute/21.04 kernel)
And found that the commit 61f879d97ce4 "powerpc/pseries: Detect secure and trusted boot state of the system" is already in (like expected, since it was upstream accepted with 5.9).
So that's good.

The CONFIG_IMA_X509_PATH is set to "/etc/keys/x509_ima.der" and CONFIG_IMA_LOAD_X509 is active ('y') for ppc64el - both was done based on LP 1643652.

And the CONFIG_SYSTEM_TRUSTED_KEYS is currently set to "".

That let's me think that there is not much to do on this particular ticket. right now.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-12-17 19:59 EDT-------
I checked out LP: #1643652. I don't know why we asked for IMA_X509_PATH there, we don't need it for OpenPower secure boot.

For guest secure boot, the end goal of my request here is to get the pieces in place to enable signed kexec - at least for the kdump case. Under secure boot, only signed kernels can be kexeced. The kernel will verify the appended signature against the .platform and .ima keyrings. The .platform keyring is fine on OpenPower secure boot and UEFI secure boot because the keyring gets populated by keys out of firmware. However, at the moment pseries firmware doesn't have any mechanism to expose trusted keys (this will change with the full key management solution!), so nothing ends up in .platform. So we want to get a key into the .ima keychain instead.

IMA_X509_PATH purports to achieve this, but unfortunately just setting it isn't quite sufficient. The key will only be loaded into the .ima keychain if it's signed by a key already trusted by the kernel, that is, in .builtin_trusted_keys.

(I discovered recently that it's also possible to load a key into .ima at runtime with keyctl, so loading the key as part of the kexec scripts could be an alternative. However, this also requires that the key being loaded into .ima be trusted by the kernel.)

So, what I think I need is:

- the key that signs the kernel on the filesystem so that it can be loaded by IMA_X509_PATH or keyctl.

- the CA key that signs the kernel-signing key to be trusted by the kernel by being included in the .pem file that gets loaded by CONFIG_SYSTEM_TRUSTED_KEYS.

[Seth Forshee (sforshee)]

I think I've got a good idea of what you're after here. Let me look into this, and I'll try to get back to you soon.

[Seth Forshee (sforshee)]

Sorry for the delayed response here, it's taken me a while to get some of the needed information.

In general this should be fine. One thing to note is that the key is self-signed, so we will need to add the signing key itself into .builtin_trusted_keys. This should still allow loading the key into the IMA keyring. It might not be necessary to do so, if IMA trusts keys in .builtin_trusted_keys for signing kexec kernels (I don't know if it does or not), but it seems to be that structuring this as though the CA and signing keys are separate keys is a good idea to ensure that this continues to work if the key setup ever changes in the future.

I'll work on getting some test packages put together in a PPA for testing. Let me know if you see any changes which need to be made as a result of the information I've provided.

[Seth Forshee (sforshee)]

Here's a test build with public cert for the opal signing key built into .builtin_trusted_keys:

https://launchpad.net/~sforshee/+archive/ubuntu/lp1903288

I'm still working out exactly how we want to distribute the key in the filesystem, but if you can try that out and let me know whether that works for you.

[Seth Forshee (sforshee)]

I should have mentioned, the kernel in comment #11 is not signed with the archive signing key since it's in a personal ppa, but the cert which is built into the kernel is for the archive key.

[Dimitri John Ledkov (xnox)]

@Daniel
"In either case, however, the CA that signs the kernel signing key needs to be built in to the kernel's .builtin_trusted_keys keyring."

On Ubuntu, for OPAL singing, on PowerPC, we do not use CA at all. It is our understanding that firmware doesn't support verifying signature chains to a CA. Thus instead we use self-signed certificates for the kernel which have not been signed by a CA.

Thus we should simply include them all in trusted keyring, and there is no need to ship anything on disk or load anything from the userspace.

We have UEFI CA which is used for UEFI booting and embedded in the UEFI shim, but I do not believe it is appropriate to use that CA here, as the revocations are controlled by a KEK key which has no relationship with POWER firmware vendors.

@sforshee

Subject: CN = Canonical Ltd. Live Patch Signing
Subject: C = GB, ST = Isle of Man, L = Douglas, O = Canonical Ltd., OU = Secure Boot, CN = "Canonical Ltd. Secure Boot Signing (POWER, 2017)"
Subject: C = GB, ST = Isle of Man, L = Douglas, O = Canonical Ltd., CN = Canonical Ltd. Kernel Module Signing

This is all that's needed for now. However, we should start also shipping the next/future OPAL signing certificate that we have generated in 2019.

Please add the 2019 opal signing certificate as debian/opal-2019-ppc64el.pem Key ID: 6B:E5:A1:25:FC:48:97:91:02:2C:2B:FB:54:91:16:F6:07:16:EA:81

There are no CA to add, and no keys to load from userspace.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-03-18 00:22 EDT-------
Apologies once again for the delay.

> @Daniel
> "In either case, however, the CA that signs the kernel signing key needs to
> be built in to the kernel's .builtin_trusted_keys keyring."
>
> On Ubuntu, for OPAL singing, on PowerPC, we do not use CA at all. It is our
> understanding that firmware doesn't support verifying signature chains to a
> CA. Thus instead we use self-signed certificates for the kernel which have
> not been signed by a CA.
>
> Thus we should simply include them all in trusted keyring, and there is no
> need to ship anything on disk or load anything from the userspace.

My mistake. Yes, if you build the kernel signing key into the trusted keyring, that should cover everything.

Kind regards,
Daniel

------- Comment From <email address hidden> 2021-03-18 00:29 EDT-------
Wait, no, hang on, this keeps tripping me up.

IMA kexec appended-signature verification uses only the %.ima and %.platform keyrings. Having the key in %.builtin_trusted_keys should logically be enough, but that's not how the code works, much to my regular frustration. Here's groovy/master security/integrity/ima/ima_appraise.c modsig_verify():

rc = integrity_modsig_verify(INTEGRITY_KEYRING_IMA, modsig);
if (IS_ENABLED(CONFIG_INTEGRITY_PLATFORM_KEYRING) && rc &&
func == KEXEC_KERNEL_CHECK)
rc = integrity_modsig_verify(INTEGRITY_KEYRING_PLATFORM,
modsig);
if (rc) {
*cause = "invalid-signature";
*status = INTEGRITY_FAIL;
} else {
*status = INTEGRITY_PASS;
}

So just having it in %:.builtin_trusted_keys doesn't suffice, we need to get it into %:.ima or %:.platform somehow.

[Dimitri John Ledkov (xnox)]

this is all very annoying! But I see what you mean now.

We probably should not add opal keys to the trusted_keyring then.

I would rather avoid introducing a new CA key whilst we cannot travel to assemble and distribute CA shards offline.

I'd rather somehow enable platform_keyring or IMA keyring, and make kernel have ability to specifies keys listed there at build time and ship the OPAL key there.

Cause the keys we use to sign kernel image & grub-image, are not the keys that are used to signed kernel modules, hence shouldn't be in the trusted kerying.

Or we can end up with a userspace .service that exports trusted_keyrings and imports them into ima keyring on everyboot. But that would be sad as well.

Let me find power machines to play around with this.

[Dimitri John Ledkov (xnox)]

Kind of wish for a config option that would do add_to_platform_keyring a built-in set of keys, until we have something like the other platforms have (ipl on s390x, uefi db on EFI platforms).

Similar to how the built-in trusted keys are initialized.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-03-18 09:39 EDT-------
(In reply to comment #22)
> Kind of wish for a config option that would do add_to_platform_keyring a
> built-in set of keys, until we have something like the other platforms have
> (ipl on s390x, uefi db on EFI platforms).
>
> Similar to how the built-in trusted keys are initialized.

Yeah, I think that might be the least-awful option. I'll see if I can bash out an "UBUNTU: SAUCE: (no-up)" patch for you :)

Kind regards,
Daniel

[Seth Forshee (sforshee)]

Sorry, I wasn't seeing emails for this bug despite being subscribed. Not sure what's going on with that.

If the key is self-signed, shouldn't having the key in .builtin_trusted_keys allow for loading it into the IMA keyring? Or is that insufficient for some reason?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-03-31 10:31 EDT-------
Hi,

> If the key is self-signed, shouldn't having the key in .builtin_trusted_keys
> allow for loading it into the IMA keyring? Or is that insufficient for some
> reason?

Yes, you could do that (I tried recently, in fact!), but then you'd need a userspace blob to perform that loading. As I understand it xnox isn't keen on more userspace blobs.

Kind regards,
Daniel

[Dimitri John Ledkov (xnox)]

BTW. is https://patchwork.kernel<email address hidden>/ of any help to us?

Ideally we'd want to do that, but load the 2017 & 2019 keys there into the .ima keyring instead of the kernel module signing ones.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-04-20 23:25 EDT-------
Hi,

Yes, actually. I've asked Nayna if she can extend those patches to also allow things to be loaded into .platform.

Kind regards,
Daniel

[Dimitri John Ledkov (xnox)]

@Nayna Jain @Daniel

Hm.... but we have CONFIG_LOAD_PPC_KEYS=y already which I would expect to be the only thing that loads keys into .platform keyring which was enabled as part of https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1866909 LTC-184073 . Which keys are present in firmware / get loaded into .platform because of that? I would have expected canonical keys to be loaded by that into the .platform keyring, or is that not the case?

Can you please share contents of "powerpc:db"? Ideally it should contain Canonical's two OPAL signing certs.

If canonical keys are not in "powerpc:db", does it make sense to then add the two Canonical keys to the .builtin_trusted_keys_keyring, and then link the whole keyring into .ima keyring?

I will attach the two Canonical OPAL signing keys here, and the ESL for them.

[Dimitri John Ledkov (xnox)]

We should not add opal keys to the built_trusted_keys_keyring as that's not the purpose of these keys. We could add them direct to .platform or .ima keyrings, but it would be best to load them from firmware direct. Are the above attached keys & ESL available from the "powerpc:db"?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-05-19 16:51 EDT-------
(In reply to comment #28)
> @Nayna Jain @Daniel
>
> Hm.... but we have CONFIG_LOAD_PPC_KEYS=y already which I would expect to be
> the only thing that loads keys into .platform keyring which was enabled as
> part of https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1866909
> LTC-184073 . Which keys are present in firmware / get loaded into .platform
> because of that? I would have expected canonical keys to be loaded by that
> into the .platform keyring, or is that not the case?

Hi,

Yes you are right that CONFIG_LOAD_PPC_KEYS enables loading of keys into .platform keyring from firmware at runtime. However, as Daniel has mentioned in his comment dated 2020-12-17 , that the .platform keyring is currently not loaded in pseries firmware as it is static keys based solution and at the moment doesn't have any mechanism to expose trusted keys (this will change with the full key management solution).

>
> Can you please share contents of "powerpc:db"? Ideally it should contain
> Canonical's two OPAL signing certs.
>

> If canonical keys are not in "powerpc:db", does it make sense to then add
> the two Canonical keys to the .builtin_trusted_keys_keyring, and then link
> the whole keyring into .ima keyring?
>
> I will attach the two Canonical OPAL signing keys here, and the ESL for them.

The final conclusion was to add a config option for PLATFORM KEYRING similar to SYSTEM_TRUSTED_KEYS mechanism. It would allow loading additional keys compiled into the kernel to be loaded only to .platform keyring. This would be in addition to the existing support for loading firmware keys at runtime on the platfom keyring. It aligns with xnox comment dated "2012-03-18".

At some point we will probably close the loop hole that allows self signed certificates loaded onto the builtin keyring to be loaded onto the IMA keyring. It's better to define a mechanism for loading additional certs on the platform keyring that would work today and will continue to work in the future.

I am supposed to start looking at the patches. I would be starting to look at them in June timeframe.

Thanks & Regards,
- Nayna

[Andrew Cloke (andrew-cloke)]

Adjusting priority to high while waiting for patches to test.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-09-10 13:31 EDT-------
I have posted the patch today - https://<email address hidden>/T/#u .

It would be very helpful if someone from Canonical can test it and confirm.

Thanks & Regards,
- Nayna

[Krzysztof Kozlowski (krzk)]

We are looking at the patches and following the upstream discussions. Once the upstream discussions have settled out, we can build a test kernel.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-10-13 16:43 EDT-------
Thanks !!

I guess a Reviewed-by or Tested-by might be helpful.

Thanks & Regards,
- Nayna

[Krzysztof Kozlowski (krzk)]

Hi Nayna,

I agree that Reviewed-by or Tested-by are in general helpful, but these tags follow strict rules in Linux kernel (see: "Reviewer's statement of oversight" in kernel documentation). I cannot provide such tags without performing review or testing. Unfortunately I cannot do the review because it is not an area of my expertise. About testing - I simply cannot test it.

Instead, you or your colleagues should engage in discussions with open-source upstream community and resolve the kbuild reported bugs and address raised concerns (about missing use-case). None of additional tags would help in avoiding doing it.

Best regards,
Krzysztof

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-10-14 11:12 EDT-------
(In reply to comment #40)
> Hi Nayna,
>
> I agree that Reviewed-by or Tested-by are in general helpful, but these tags
> follow strict rules in Linux kernel (see: "Reviewer's statement of
> oversight" in kernel documentation). I cannot provide such tags without
> performing review or testing. Unfortunately I cannot do the review because
> it is not an area of my expertise. About testing - I simply cannot test it.

Thanks Krzysztof for your response. Of course, you should give Reviewed-by and Tested-by only if you actually do it.

Sorry, if I assumed the details here. When I asked for the tags, what I meant was that if someone from Canonical can actually review and test the patch and then send their Reviewed-By, Tested-by. Sorry for the misunderstanding and I hope it is clarified now.

In case it helps, I guess it should be possible to test it on any x86/Power system by embedding your own generated key in .platform keyring using this patch and ensure verified kexec on secure boot enabled system.

>
> Instead, you or your colleagues should engage in discussions with
> open-source upstream community and resolve the kbuild reported bugs and
> address raised concerns (about missing use-case). None of additional tags
> would help in avoiding doing it.
>

v3 version of the patch has addressed all the concerns which are raised till now.

https://<email address hidden>/

Thanks & Regards,
- Nayna

[Frank Heimes (fheimes)]

I just 'extracted' the patch from the upstream v3 discussion thread and attach it here.
(Even if there seems to be a request to split it into two patches, but that would probably have no functional impact.)

[Frank Heimes (fheimes)]

Hi @Nayna even if it looks like your patch (v3) is still under discussion on the mailing list (https://lore.kernel.org<email address hidden>/)
we've built a test kernel that allows to try and test what you already have (v3) in an Ubuntu kernel (5.15.0-9.9) context:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1903288/

Not sure if your focus is still on this v3 or if you already proceeded ...

[Dimitri John Ledkov (xnox)]

Added my own review https://lore<email address hidden>/

A few optional comments; and one required change needed to add one more ifdef.

[Frank Heimes (fheimes)]

I've just noticed Nayna's v4 (from Nov 11th):
https://<email address hidden>/
"
v4:
* Split into two patches as per Mimi Zohar and Dimitri John Ledkov
recommendation.
"

[Frank Heimes (fheimes)]

I've just 'extracted' the v4 patch set from the upstream mailing-list and attach it here.
(builds are ongoing ...)

[Frank Heimes (fheimes)]

I just kicked off another build for a new patched kernel that allows to try and test the v4 patch-set in an Ubuntu kernel 5.15.0-9.9 context:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1903288-v4/
(will take a while until it's completed ...)

[Frank Heimes (fheimes)]

I've just 'extracted' the v5 patch set from the upstream mailing-list and attach it here.
(builds are available: https://launchpad.net/~fheimes/+archive/ubuntu/lp1903288-v5/ )

[Frank Heimes (fheimes)]

Over the year break a v7 was made available and was discussed at the mailing list:
https://<email address hidden>/

Since this mainly seem to have structural changes (patch split etc.) and comment changes compared to the previous versions and since there is a request to expand/rework the description a bit more, I'll wait for a version > v7 to test build again in the PPA.

I hope that we are close to a version that will be recommended for upstream acceptance, so that we can take it from there and consider integration into the Ubuntu kernel.

[bugproxy (bugproxy)]

(builds are ongoing ...)

[Frank Heimes (fheimes)]

meanwhile v8 became available:
* Includes Jarkko's feedback on patch description and removed Reported-by for Patch 1
The extracted v8 patch-set is attached.
Builds are currently running and are soon available here:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1903288-v8/

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Frank Heimes (fheimes)]

Planned new target is 24.04.

[Frank Heimes (fheimes)]

Closing in favor of a solution based on dynamic keys.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-03-12 17:34 EDT-------
Closing on our side as ALT_SOLUTION_AVAIL.