GRUB menu loading failure via HTTP Boot on BlueField

Bug #2043084 reported by Chengen Du
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
grub2-unsigned (Ubuntu)
Status tracked in Noble
Focal
New
Undecided
Mate Kukri
Jammy
In Progress
Undecided
Mate Kukri
Mantic
Invalid
Undecided
Mate Kukri
Noble
Fix Released
Undecided
Mate Kukri

Bug Description

[Impact]
The GRUB menu fails to load via HTTP boot but functions properly with PXE boot, displaying the error message:
error: Fail to send a request! status=0x8000000000000002.

The error code 0x8000000000000002 corresponds to GRUB_EFI_INVALID_PARAMETER.
Notably, this issue is unreproducible in QEMU but surfaces on BlueField.

[Fix]
Bisecting Oracle's patches, we confirmed with the customer that the issue was resolved with the following commit:
 https://github.com/rhboot/grub2/commit/205b7b44bb2a7911f1e8c1ce22d66c111a4b429e

[Test Plan]
We've requested the customer to test the patch on BlueField.
Our test plan will focus on checking for any regressions after applying this patch.

1. Run a DHCP server on the host and configure /etc/dhcp/dhcpd.conf with the provided content.
...
host http_boot {
  option host-name "http_boot";
  hardware ethernet XX:XX:XX:XX:XX:XX;
  fixed-address 192.168.122.10;
  class "HTTPClient" {
    match if substring (option vendor-class-identifier, 0, 10) = "HTTPClient";
    option vendor-class-identifier "HTTPClient";
    filename <-efi's path->;
  }
}
...
2. Start a UEFI VM and use "virsh console" for access.
3. Execute "systemctl reboot --firmware-setup" in the console to enter the UEFI interface.
4. Test the successful loading of the GRUB menu by selecting "Boot Manager" -> "UEFI HTTPv4."

[Where problems could occur]
The patch only updates a member variable type to comply with UEFI 2.9, preserving the original behavior.
In QEMU, using EFI v2.70 from EDK II, we can still access the GRUB menu.

Related branches

CVE References

Chengen Du (chengendu)
Changed in grub2 (Ubuntu):
assignee: nobody → Chengen Du (chengendu)
Revision history for this message
Julian Andres Klode (juliank) wrote :

Reassigning to boot engineering. BlueField seems to be arm64? I reassigned this to grub2-unsigned accordingly.

This is not something you should be touching yourselves, really.

Changed in grub2 (Ubuntu):
status: New → Triaged
assignee: Chengen Du (chengendu) → nobody
affects: grub2 (Ubuntu) → grub2-unsigned (Ubuntu)
Changed in grub2-unsigned (Ubuntu):
assignee: nobody → Mate Kukri (mkukri)
tags: added: foundations-todo
Revision history for this message
Chengen Du (chengendu) wrote :

Bluefield is arm64, and the patch has undergone successful testing.
I've already initiated a bug report in grub2-unsigned. Should this be marked as a duplicate?
The bug report is available at: https://bugs.launchpad.net/ubuntu/+source/grub2-unsigned/+bug/2043085.
Additionally, I'm inquiring whether I should provide the debdiff for this issue or await action from the boot engineering.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Mate is merging the fixes with some others and we should have an update available in the next days.

Revision history for this message
Mate Kukri (mkukri) wrote (last edit ):

@chengendu https://code.launchpad.net/~mkukri/grub/+git/grub/+merge/455398 (sorry first MP had a typo'd branch name)

Revision history for this message
Julian Andres Klode (juliank) wrote :

Marking mantic as invalid because it doesn't have the http code yet, it may get it back with noble, in which case the bug will become fix comitted/released again to make it clear that the re-added network patch set has the fix.

Changed in grub2-unsigned (Ubuntu Jammy):
assignee: nobody → Julian Andres Klode (juliank)
assignee: Julian Andres Klode (juliank) → Mate Kukri (mkukri)
status: New → Fix Committed
status: Fix Committed → In Progress
Changed in grub2-unsigned (Ubuntu Mantic):
status: New → Invalid
Mate Kukri (mkukri)
Changed in grub2-unsigned (Ubuntu Focal):
assignee: nobody → Mate Kukri (mkukri)
Changed in grub2-unsigned (Ubuntu Mantic):
assignee: nobody → Mate Kukri (mkukri)
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (5.5 KiB)

This bug was fixed in the package grub2-unsigned - 2.12~rc1-12ubuntu2

---------------
grub2-unsigned (2.12~rc1-12ubuntu2) noble; urgency=medium

  * Merge from Debian unstable; remaining changes:
    - Add Ubuntu sbat data
    - build-efi-images: do not produce -installer.efi.signed. LP: 1863994
    - grub-common: Install canonical-uefi-ca.crt
    - Check signatures
    - Support installing to multiple ESP (LP: 1871821)
    - Disable various bits on i386
    - Split out unsigned artefacts into grub2-unsigned
    - Vcs-Git: Point to ubuntu packaging branch
    - Relax dependencies on grub-common and grub2-common
    - grub-pc: Avoid the possibility of breaking grub on SRU update due
      to ABI change
    - UBUNTU: Default timeout changes
    - Revert "Add jfs module to signed UEFI images. Closes: #950959"
    - Revert "Add f2fs module to signed UEFI images"
    - Install grub-initrd-fallback.service again
    - Build using -O1 on s390x to avoid misoptimization
    - grub-check-signatures: Support gzip compressed kernels (LP: #1954683)
    - grub-multi-install: Reset partition type between partitions (LP: #1997795)
    - Drop i386 from grub-efi-amd64* (LP: #2020907)
    - Turn depends on grub-efi-amd64/arm64 unversioned
    - forward port fix for LP: #1926748
    - Make the grub2/no_efi_extra_removable setting work correctly
    - Forward port the fix for LP: #1930742 and make it conditional (xenial/bionic only)
    - Build grub2-unsigned packages with xz compression
    - Revert: "Have -bin packages Break pre-2.12 -signed packages.", this is not
      compatible with our versioning schemes.
    - Install a /usr/lib/grub/grub-sort-version and use that to sort versions as
      it respects GRUB_FLAVOUR_ORDER. Depend on python3 to do so.
    - rules: Add DPKG_BUILDPACKAGE_OPTIONS to generate-grub2-unsigned
    - Replaced patches:
      - installe-signed.patched
      - grub-install-extra-removable.patch
      - grub-install-removable-shim.patch
    - Added patches:
      + rhboot-f34-dont-use-int-for-efi-status.patch
      + rhboot-f34-make-exit-take-a-return-code.patch
      + suse-grub.texi-add-net_bootp6-document.patch
      + ubuntu-add-devicetree-command-support.patch
      + ubuntu-add-initrd-less-boot-fallback.patch
      + ubuntu-add-initrd-less-boot-messages.patch
      + ubuntu-boot-from-multipath-dependent-symlink.patch
      + ubuntu-dont-verify-loopback-images.patch
      + ubuntu-fix-lzma-decompressor-objcopy.patch
      + ubuntu-grub-install-extra-removable.patch
      + ubuntu-install-signed.patch
      + ubuntu-mkconfig-leave-breadcrumbs.patch
      + ubuntu-os-prober-auto.patch
      + ubuntu-recovery-dis_ucode_ldr.patch
      + ubuntu-resilient-boot-boot-order.patch
      + ubuntu-resilient-boot-ignore-alternative-esps.patch
      + ubuntu-shorter-version-info.patch
      + ubuntu-speed-zsys-history.patch
      + ubuntu-support-initrd-less-boot.patch
      + ubuntu-verifiers-last.patch
      + ubuntu-zfs-enhance-support.patch
      + ubuntu-zfs-gfxpayload-dynamic.patch
      + ubuntu-zfs-gfxpayload-keep-default.patch
      + ubuntu-zfs-insmod-xzio-and-lzopio-on-xen.patch
      + ubuntu-zfs-mkconfig-recovery-title.patch
     ...

Read more...

Changed in grub2-unsigned (Ubuntu Noble):
status: Triaged → Fix Released
Revision history for this message
Chengen Du (chengendu) wrote :

@mkukri, I apologize for my lack of familiarity with the grub EFI release process. Could you confirm if the arm64 testing version will be available for download at http://ports.ubuntu.com/ubuntu-ports/dists/?

Revision history for this message
Mate Kukri (mkukri) wrote (last edit ):

Hi @chengendu,

For Noble, I believe it already is available there.

Mantic is not affected as it is missing the patches needed for UEFI HTTP booting (might be SRUd later).

We've decided against spending time fixing Lunar unless it is known to be required.

Jammy and Focal have been signed and waiting to be released. I believe they will appear in the proposed pocket after that and will get eventually promoted to updates.

Revision history for this message
Julian Andres Klode (juliank) wrote (last edit ):

For stable releases, I'm unembargoing the signed binaries now, they'll be available in

https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/proposed-public/

shortly (well mantic and jammy are, focal is missing).

After that

- I'll copy the binary to the main archive
- SRU team accepts it into -proposed
- Verification needs to be done and it needs to age 7 days
- SRU team releases it to -updates

Revision history for this message
Steve Langasek (vorlon) wrote :

> The patch only updates a member variable type to comply with UEFI 2.9, preserving the original behavior.

Please explain what "preserving the original behavior" is supposed to mean here. You are clearly changing the size of a struct element on all 64-bit architectures, and this struct is being passed to a UEFI function. So how can unconditionally changing the size of the struct elements, and the offsets of all elements after it in the struct, "preserve the original behavior"?

Changed in grub2-unsigned (Ubuntu Jammy):
status: In Progress → Incomplete
Revision history for this message
Mate Kukri (mkukri) wrote :

@vorlon I'll have to cross-reference more spec versions next week, but I suspect what's going in is the size of those fields were incorrect, but it happened to sometimes work (when the allocated memory didnt contain garbage) due to the offsets matching because of C struct packing rules adding padding after the 32-bit ints because they are followed by 64-bit pointers (and on 32-bit arches this is a no-op change).

Revision history for this message
Mate Kukri (mkukri) wrote :

@vorlon The upstream commit message is confusing and incorrect, what this patch actually does is fix the type of those fields that were always incorrect in GRUB.

I have cross-referenced UEFI specification starting from the introduction of HTTP boot, and I can confirm those fields always had type UINTN. These all have type UINTN in these fields:
- UEFI 2.5 (this seems to be the version that specificed HTTP boot first)
- UEFI 2.8
- UEFI 2.9
- and the current UEFI 2.10.

And I suppose the reason this wasn't caught earlier is because as I have mentioned yesterday, C struct packing rules made the offset of fields still correct despite them being the wrong size. However 64-bit firmware still read them as 64-bit qwords and I guess if the padding contained garbage it would cause bad behavior.

Revision history for this message
Steve Langasek (vorlon) wrote :

Can the test plan please be extended to include tests on hardware other than the customer hardware?

Revision history for this message
Mate Kukri (mkukri) wrote :

@vorlon I have confirmed that this fixes UEFI HTTP boot on arm64 qemu too, and can repeat the same test for SRU verification.
I don't have any UEFI native arm64 hardware to test with however.

Revision history for this message
Mate Kukri (mkukri) wrote :

@vorlon Spoke to dannf, and there is a plan now for testing this on hyperscale lab machines.

Mate Kukri (mkukri)
Changed in grub2-unsigned (Ubuntu Jammy):
status: Incomplete → In Progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.