encounter general protection fault while pxe booting from MaaS server

[sandeep kumar pabboj]

[Impact]
grub can cause a general protection fault in EFI by freeing the wrong number of bytes for an address it allocated, causing the machine not to boot.

[Test plan]
1. Have a Dell 14G server with ubuntu OS and MAAS 2.9.2 application installed. Configure MaaS Deploy Settings to Centos7.
2. Configure another Dell 14G SUT with PXE on boot and enlist it in MAAS.
3. Deploy Centos7 in SUT through MaaS.
4. Power off the system after Centos deployment.
5. Power on the system, SUT goes to MaaS-PXE and we now should be able to boot to installed Centos-7 (Issue is RSOD here).

[Where problems could occur]
We could free too little or too much memory if the code is wrong? Basically all this does is change a free() call to use sizeof() instead of a hardcoded value that was used previously to allocate the buffer. Which should lead to the same result as now in the worst case.

[Original bug report]
Observing RSOD when ubuntu installed server setup with MaaS application as per below procedure and pointing out different OS images from grub to deploying OS for connected server.

certification-static.canonical.com/docs/MAAS_Advanced_Network_Installation_And_Configuration.pdf)

Steps to reproduce the issue,
1. Have a server with ubuntu OS and MAAS 2.9.2 application installed

2. Make sure remote server is configured with PXE and enlisted into MAAS application.

[In remote server enable PXE boot in the BIOD network settings and also configure uEFI boot mode and PXE boot device in BIOS settings]

4. Go to MAAS server and use curl command to check the current grub configuration for new enlisted server and make sure current grub configuration mapping to the "MAC address of NIC which is pointing to the CentOS image to deploy or to install into remote server which is enlisted into MAAS. [please refer "grub_commission" snapshot for more details]

5. After above check power on the new enlisted server and run the PXE boot via NIC port.
RSOD should be able to triggered at stage of "booting under maas direction.." [please refer "RSOD_error" snapshot for mroe details]

6. In step 4, if groub config for commissioned machine is pointing to ubuntu boot kernel the RSOD wont be triggered.

[sandeep kumar pabboj (sandeep-kumar-pabboj)]

As per our internal evaluation, seems issue is in GRUB utility used by MAAS (2.9.2).

In the source code of grub(../grub-core/loader/i386/efi/linux.c) at line #215 "gub_efi_allocate_pages_max" allocates ONE PAGE but while freeing the pages in line #365 FOUR PAGES are got freed in "grub_efi_free_pages",

/*Line 215*/ params = grub_efi_allocate_pages_max (0x3fffffff,
/*Line 216*/ BYTES_TO_PAGES(sizeof(*params)));
/*Line 365*/ grub_efi_free_pages ((grub_efi_physical_address_t)(grub_addr_t)params,
/*Line 366*/ BYTES_TO_PAGES(16384));

And RSOD is not observed after changing the above code as like below in line #366,
/* Line 365*/ grub_efi_free_pages ((grub_efi_physical_address_t)(grub_addr_t)params,
/* Line 366*/ BYTES_TO_PAGES(sizeof(*params));

Kindly help us verifying and confirming above observations.

[Jeff Lane  (bladernr)]

what version of grub is being used?

Also, is the reproducible using MAAS 3.1?

[sandeep kumar pabboj (sandeep-kumar-pabboj)]

Hi Jeff,

Yes with MAAS 3.1 also we see the same issue.

grub version is "grub-efi-amd64-2.04-1ubuntu44.2-grubnetx64".

[Jeff Lane  (bladernr)]

Thanks for trying that Sandeep. We'll have to escalate that internally and see if we can get someone to take a deeper look.

[sandeep kumar pabboj (sandeep-kumar-pabboj)]

@jeff lane
please let us know if any updates on this issue.

[Jeff Lane  (bladernr)]

Ack... Michael is going to refer this to the teams that manage grub and the installer to see how they want to proceed.

[Michael Reed (mreed8855)]

I have forwarded this bug internally to the appropriate teams and it is under review.

[Julian Andres Klode (juliank)]

This should be fixed in grub 2.06-2ubuntu5 which will land in jammy in the next couple days - it's currently in proposed, but it takes a while for the signed images to be accepted.

We should still SRU that to bionic and focal - after the point release

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.06-2ubuntu5

---------------
grub2 (2.06-2ubuntu5) jammy; urgency=medium

  [ Julian Andres Klode ]
  * Free correct size when freeing params, rather than 16 Ki (LP: #1958623)
  * Build with FUSE3 (LP: #1935659)
  * Only run os-prober on first run and if it previously found other OS
    (LP: #1955109)

  [ Heinrich Schuchardt ]
  * Rename grub-core/loader/efi/linux.c
  * Add patches for GRUB on RISC-V
  * fat: fix listing the root directory
  * Enable building for RISC-V (LP: #1876620)

  [ Julian Andres Klode ]
  * Re-enable peimage code on other archs outside secure boot; this
    fixes LP: #1947046 when not booting in secure boot mode (secure
    boot pending security review of the code)

 -- Julian Andres Klode <email address hidden> Fri, 18 Feb 2022 17:21:16 +0100

[Jeff Lane  (bladernr)]

@julian I added focal and bionic tasks per your comment #10, should we rewrite this as an SRU bug now, or would you ratner a separate SRU bug to handle focal and bionic?

[Julian Andres Klode (juliank)]

No, we'll use this bug, but the tasks need to move to grub2-unsigned (where we build the supported UEFI binaries)

[Julian Andres Klode (juliank)]

Updated the description to avoid ominous acronyms and describe the issue at hand (TIL RSOD = red screen of death).

[Julian Andres Klode (juliank)]

Since we want this SRUed I'm going to need someone who can update the bug with a test case for the SRU and then do the testing of the SRUs in focal and bionic; as I don't think the original bug report is universally reproducible.

[Sujith Pandel (sujithpandel)]

Test case steps:

1. Have a Dell 14G server with ubuntu OS and MAAS 2.9.2 application installed. Configure MaaS Deploy Settings to Centos7.
2. Configure another Dell 14G SUT with PXE on boot and enlist it in MAAS.
3. Deploy Centos7 in SUT through MaaS.
4. Power off the system after Centos deployment.
5. Power on the system, SUT goes to MaaS-PXE and we now should be able to boot to installed Centos-7 (Issue is RSOD here).

We can test the fix once it is made available.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14

---------------
grub2-unsigned (2.06-2ubuntu14) kinetic; urgency=medium

  * SECURITY UPDATE: Fix out of bounds writes due specially crafted fonts.
    - add debian/patches/font-Fix-several-integer-overflows-in-grub_font_construct.patch
    - add debian/patches/font-Fix-an-integer-underflow-in-blit_comb.patch
    - CVE-2022-2601, CVE-2022-3775
    - LP: #1996950
  * Fix various issues as a result of fuzzing, static analysis and code
    review:
    - add debian/patches/font-Reject-glyphs-exceeds-font-max_glyph_width-or-font-m.patch
    - add debian/patches/font-Fix-size-overflow-in-grub_font_get_glyph_internal.patch
    - add debian/patchces/font-Remove-grub_font_dup_glyph.patch
    - add debian/patches/font-Fix-integer-overflow-in-ensure_comb_space.patch
    - add debian/patches/font-Fix-integer-overflow-in-BMP-index.patch
    - add debian/patches/font-Fix-integer-underflow-in-binary-search-of-char-index.patch
    - add debian/patches/fbutil-Fix-integer-overflow.patch
    - add debian/patches/font-Harden-grub_font_blit_glyph-and-grub_font_blit_glyph.patch
    - add debian/patches/font-Assign-null_font-to-glyphs-in-ascii_font_glyph.patch
    - add debian/patches/normal-charset-Fix-an-integer-overflow-in-grub_unicode_ag.patch
  * Enforce verification of fonts when secure boot is enabled:
    - add debian/patches/kern-efi-sb-Enforce-verification-of-font-files.patch
  * Bundle unicode.pf2 in a squashfs memdisk attached to the signed EFI binary
    - update debian/control
    - update debian/build-efi-image
    - add debian/patches/font-Try-opening-fonts-from-the-bundled-memdisk.patch
  * Fix LP: #1997006 - add support for performing measurements to RTMRs
    - add debian/patches/commands-efi-tpm-Refine-the-status-of-log-event.patch
    - add debian/patches/commands-efi-tpm-Use-grub_strcpy-instead-of-grub_memcpy.patch
    - add debian/patches/efi-tpm-Add-EFI_CC_MEASUREMENT_PROTOCOL-support.patch
  * Fix the squashfs tests during the build
    - remove debian/patches/ubuntu-fix-reproducible-squashfs-test.patch
    - add debian/patches/tests-Explicitly-unset-SOURCE_DATE_EPOCH-before-running-f.patch
  * Bump SBAT generation:
    - update debian/sbat.ubuntu.csv.in
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Chris Coulson <email address hidden> Wed, 16 Nov 2022 14:40:42 +0000