[24.04] Power guest secure boot with static keys: GRUB2 portion

[bugproxy]

Due to the fact that the work and backlog on grub2 in jammy is pretty big and the complexity high, I discussed with juliank (Foundations) the other day, and the outcome was to transfer this ticket into a FFe, just to be prepared.

The patch set that is in discussion here is especially for Ubuntu secure boot on PowerVM .
PowerVM is mandatory for the most recent Power 10 family, and Power 10 support is planned for Ubuntu Server 22.04 (and is customer driven).
Since it's uncertain if a re-worked patch can be picked-up in time for the jammy Feature Freeze on Feb. the 24th, I want to raise the attention of the Ubuntu Release Team,
and kindly ask for acknowledge.

Required tests on this will be done with the help of the customer/partner IBM.
_________________________

== Comment: #2 - Daniel John Axtens <email address hidden> - 2020-11-05
19:50:03 ==
This bug covers the grub part of our LPAR Secure Boot design, which I'd like to propose for your 21.04 release.

Background
==========

Please find attached an overall outline of the design.

Patches
=======

I've posted the 3 main parts of this feature to the grub mailing list:

 - Allowing grub to be signed with an appended signature to be verified by firmware: https://lists.gnu.org/archive/html/grub-devel/2020-08/msg00037.html
 - Teaching grub how to verify appended signatures on Linux kernels: https://lists.gnu.org/archive/html/grub-devel/2020-10/msg00152.html
 - Linking grub's verification to the secure boot status from firmware: https://lists.gnu.org/archive/html/grub-devel/2020-10/msg00048.html

(We do need one other piece just to increase the amount of memory grub allocates for itself in as it starts. We have a couple of prototypes, one which we've posted to the list and one which we haven't. The one we've posted breaks booting of guests with 512MB of memory, which - as far as I can tell - is still supported by Ubuntu, and indeed is the default for guests created by uvtool. So we'll post an alternative to the list shortly.)

Because of the grub development cycle, these patches will not be merged upstream for 2.06. We are hoping, given the incredibly slow pace of grub development and the number of out-of-tree patches that have traditionally been required for UEFI secure-boot support, that you'd be willing to carry these out of tree while we try to have them merged.

Next Steps
==========

We need the following information from you:

 - If you're willing to take grub changes, what tree would you like us to backport our grub changes onto? I imagine it's just the development tree on launchpad, but I thought I'd check.

 - What keys would you like to use? As we are using static keys, we will need to embed an x509 certificate for verifying grub in firmware. We will need a production certificate, and we can also support test keys while firmware is still in development. Either you can supply a test key or you can use the existing 'imprint' keys that we are using. (Neither will be supported in production firmware builds - this is just for before we hit production firmware levels.)

 - The addition of a signed binary will make installing grub more complicated, as grub-install (out of the box) will create and install an unsigned binary. There's also likely implications for the installer. What parts of this process do you need us to develop?

Building, Signing and Testing
=============================

If you would like to, you can experiment with this already using qemu and SLOF, although our production solution will only support PFW. Details are on the list: https://lists.gnu.org/archive/html/grub-devel/2020-10/msg00048.html

A slightly more involved process is required for a fully generic distro-friendly solution, we can share more details on that.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Frank Heimes (fheimes)]

Just cross-referencing, this is the kernel part: LP 1903288

[Dimitri John Ledkov (xnox)]

Note that we currently ship a subset of rhboot/grub secureboot patches for EFI x86_64, modified to support ARM64.

The packaging of them is a bit convoluted, given that they continuously are getting reviewed and have not yet been merged upstream despite being 9 years old.

I hope it will be quicker to get powerpc patches into upstream, however, it will take time for them to get security review and accepted. Until they are accepted, please maintain public git repository from which we can pull in the patches. Ideally, with branches, per upstream tag, rebased and maintained on ongoing basis from until, and indefinitely. Unless of course they make into upstream.

Ubuntu's packaging, with patches applied is maintained at https://code.launchpad.net/~ubuntu-core-dev/grub/+git/ubuntu you could choose to push to launchpad too, and rebase onto "ubuntu" branch which is always the packaging for the current devel series. At the moment that is hirsute. Once series goes stable named branches appear, i.e. "focal". Those branches look very odd, because they have debian/patches that have been serialized from git commits that have been merged already using git-dpm. Thus the tree has many merged commits, but the code after git clone should look as the source tree we are building. Thus ideally / hopefully, your patches rebase/merge onto "ubuntu" branch cleanly. I did not try this yet.

You must on the ongoing basis support enrolling arbitrary test key, and using it for testing. Launchpad has PPA archives, which allow Ubuntu developers build and sign packages. These archives create on-the-fly signing keys and allow one to upload and sign new experimental grub builds, but also kernels. Similar to how we continuously ask for testing secure-boot/lock-down of kernel signed with ephemeral test keys; we will do so with grub as well. Similarly you too, can upload packages to ubuntu PPA with a copy of a kernel, to observe it getting signed and published as debs with signed bootloader & kernel and the details of the signing key. Once all the necessary packaging is in place. We already have this requirement for the Power kernel enablement, however currenty the turn-around times of testing that are very long and have almost breached the timelines for new kernel uploads. If grub/kernel development is unreasonably delayed on testing new releases, we would be turning off signing with production keys in order to avoid revocation event.

Does existing / old firmware handle signed grub binaries just fine?

Ideally, we prefer to always use and install signed binaries. Regardless if secureboot is on, or off. Such that users can migrate disks and enable secureboot at a later point. Failing that, we'd need something similar to --uefi-secure-boot flag in grub-install to control and be sensitive to signed bits. But if you notice for x86 today we always check for signed grub and install that, in preference to installing any other images.

For the installer .iso - is there any signing required there, or will the .iso images remain unchanged?

I will try to prepare sample test packages. However it is not quite so quick. This will not be done instantly.

W.r.t. grub2 versions you will obser...

[Dimitri John Ledkov (xnox)]

As a brand new feature, this cannot be critical, as existing hardware doesn't already support or require grub secureboot. Priority set to wishlist, as it is a new feature request.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-12-04 07:52 EDT-------
Hi,

Sorry for the delay in getting back to you.

> Note that we currently ship a subset of rhboot/grub secureboot patches for
> EFI x86_64, modified to support ARM64.
>
> The packaging of them is a bit convoluted, given that they continuously are
> getting reviewed and have not yet been merged upstream despite being 9 years
> old.

Right. It is indeed frustrating! I have successfully transplanted my patches onto a slightly older RHEL tree so it sounds like your trees shouldn't be too much extra work.

> I hope it will be quicker to get powerpc patches into upstream, however, it
> will take time for them to get security review and accepted. Until they are
> accepted, please maintain public git repository from which we can pull in
> the patches. Ideally, with branches, per upstream tag, rebased and
> maintained on ongoing basis from until, and indefinitely. Unless of course
> they make into upstream.
>
> Ubuntu's packaging, with patches applied is maintained at
> https://code.launchpad.net/~ubuntu-core-dev/grub/+git/ubuntu you could
> choose to push to launchpad too, and rebase onto "ubuntu" branch which is
> always the packaging for the current devel series. At the moment that is
> hirsute. Once series goes stable named branches appear, i.e. "focal". Those
> branches look very odd, because they have debian/patches that have been
> serialized from git commits that have been merged already using git-dpm.
> Thus the tree has many merged commits, but the code after git clone should
> look as the source tree we are building. Thus ideally / hopefully, your
> patches rebase/merge onto "ubuntu" branch cleanly. I did not try this yet.
>

I have some limited experience with debian packaging - I used to work in your Sustaining Engineering team, but mostly on the kernel. I'll try to get myself set up on launchpad shortly.

> You must on the ongoing basis support enrolling arbitrary test key, and
> using it for testing. Launchpad has PPA archives, which allow Ubuntu
> developers build and sign packages. These archives create on-the-fly signing
> keys and allow one to upload and sign new experimental grub builds, but also
> kernels. Similar to how we continuously ask for testing
> secure-boot/lock-down of kernel signed with ephemeral test keys; we will do
> so with grub as well. Similarly you too, can upload packages to ubuntu PPA
> with a copy of a kernel, to observe it getting signed and published as debs
> with signed bootloader & kernel and the details of the signing key. Once all
> the necessary packaging is in place. We already have this requirement for
> the Power kernel enablement, however currenty the turn-around times of
> testing that are very long and have almost breached the timelines for new
> kernel uploads. If grub/kernel development is unreasonably delayed on
> testing new releases, we would be turning off signing with production keys
> in order to avoid revocation event.

So this might be a bit challenging - our entire secure boot architecture is currently built around the idea of static keys embedded in firmware for verifying grub. The next step for us after this is ...

[Dimitri John Ledkov (xnox)]

We have been reworking grub packaging in Ubuntu which will be released soon.

One of the features is to split src:grub2 into src:grub2 and src:grub2-unsigned, such that we can build -bin -dbg & -signed packages separately from all other ones. This will allow us to build signed grub modules for a given platform once, and binary-copy them across releases. In theory this might allow us to bring back grub secureboot on power to any past series, if of course everything else integrates well with it.

I did see the backports that you have provided for the rhel tree and for next upstream release. I will look into integrating those and providing some test builds, once this next set of grub uploads is out.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-03-01 19:36 EDT-------
(In reply to comment #10)
> We have been reworking grub packaging in Ubuntu which will be released soon.
>
> One of the features is to split src:grub2 into src:grub2 and
> src:grub2-unsigned, such that we can build -bin -dbg & -signed packages
> separately from all other ones. This will allow us to build signed grub
> modules for a given platform once, and binary-copy them across releases. In
> theory this might allow us to bring back grub secureboot on power to any
> past series, if of course everything else integrates well with it.
>
> I did see the backports that you have provided for the rhel tree and for
> next upstream release. I will look into integrating those and providing some
> test builds, once this next set of grub uploads is out.

Hi,

Firstly, sincere apologies for the delay in getting back to you.

That sounds like a good plan.

I will be posting a fresh version of the series to the mailing list in the next week or so - certainly not until after 2 March. I'll sync up on backports after that.

Kind regards,
Daniel

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-03-30 10:49 EDT-------
Hi,

I've just posted my 2.06~rc1 version: https://lists.gnu.org/archive/html/grub-devel/2021-03/msg00359.html

I have forked https://code.launchpad.net/~ubuntu-core-dev/grub/+git/ubuntu/+ref/ubuntu on launchpad and am cherry-picking my changes now. I'll let you know how I get on.

Kind regards,
Daniel

[Frank Heimes (fheimes)]

Hi Daniel,
well, we obviously didn't matched the timeline discussed above.
However, there is now a 2.06 in jammy (22.04), hence I wanted to quickly check you where we are?
I couldn't find your patches in 2.06 upstream yet, are you still pursuing to bring them upstream?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2022-01-12 00:18 EDT-------
Oof, sorry to have let this slip for so long, I've been totally swamped.

The patches have got backed up behind some broader work in the grub community for refactoring how grub deals with things that require larger memory allocations. (Currently grub is limited to 32MB on power, which is not enough to verify the signature on a modern kernel in the current implementation.)

I could provide a backport to Jammy if that's something Ubuntu would be open to accepting.

Kind regards,
Daniel

[Frank Heimes (fheimes)]

Hi Daniel,
yes, please provide a backport to jammy.
That is the only change (though not a confirmation) to get that in.
Thx

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2022-02-07 19:02 EDT-------
Hi,

Apologies for the delay and thanks for your patience.

I haven't been able to properly test it yet, but some backports that at least compile and pass the relevant parts of the test suite can now be found at https://git.launchpad.net/~daxtens/ubuntu/+source/grub2/log/?h=ubuntu-appendedsig-2.11

Hopefully - hopefully! - I will be able to test it more this week.

If it's helpful to the team I'm happy to make that fit the Ubuntu packaging system a bit more closely - I know the basic principles (debian/patches, debian/patches/series, etc) but I'm not very good with the git-buildpackage tools just yet.

One thing that we've hit in other contexts is that this complicates the grub 'prefix'/search path: that is, how grub finds the grub.cfg file. Previously, this was determined by grub-install and encoded in the elf file, but that doesn't work for us because the whole elf file has to be signed at build time, so it can't have system-specific info embedded. [On UEFI, this problem doesn't arise because the Efi System Partition has a fat filesystem so it's easy to dump system-specific paths into a file on the ESP.]

Currently the backports do a search, which does work but is potentially a bit slow and error-prone. You simply build grub with a prefix representing just the path, and it goes and looks for the relevant partition. So if you have /boot/grub/grub.cfg and /boot is its own partition, you would build with "-p /grub"

Another external team is working on a way to put a grub-env block at the end of the PReP partition which could be a more elegant solution. I will let you know how everyone gets on there. In the mean time, if you have any input please let me know.

Kind regards, and once again my sincere apologies for the massive delays,
Daniel

Kind regards,
Daniel

[Frank Heimes (fheimes)]

Withdrawing as FFe (and unsubscribing the Ubuntu Release Team), since it will not arrive in time.
New plan is to target this for 22.10, with the option to potentially get it in to 22.04.1.

[Launchpad Janitor (janitor)]

[Expired for grub2 (Ubuntu) because there has been no activity for 60 days.]

[Frank Heimes (fheimes)]

Planned new target is 24.04.

[Frank Heimes (fheimes)]

Closing in favor of a solution based on dynamic keys.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-03-12 17:33 EDT-------
Closing on our side as ALT_SOLUTION_AVAIL.