QEMU UEFI VMs fail to request the /grub/grub.cfg over HTTP when booting with UEFI HTTP

[Blake Rouse]

While implementing the complete UEFI HTTP booting path for MAAS I have hit a roadblock. Grub2 is loaded but then fails to request the /grub/grub.cfg as it does when booting with UEFI PXE (aka. TFTP).

The dhcpd.conf has the following in it to tell the UEFI firmware to load the bootx64.efi (signed-shim) over HTTP.

} elsif option arch = 00:0f {
    # uefi_amd64_http
    filename "http://192.168.122.2:5248/bootx64.efi";
    option vendor-class-identifier "HTTPClient";
} elsif option arch = 00:10 {
    # uefi_amd64_http
    filename "http://192.168.122.2:5248/bootx64.efi";
    option vendor-class-identifier "HTTPClient";
} elsif option arch = 00:0B {

This works correctly because the UEFI firmware does load the bootx64.efi then the shim loads the grubx64.efi over HTTP.

2019-03-27 23:21:59 provisioningserver.rackdservices.http: [info] bootx64.efi requested by 192.168.122.193
2019-03-27 23:21:59 provisioningserver.rackdservices.http: [info] grubx64.efi requested by 192.168.122.193

No other requests are received from that point forward. Grub2 reports the following error:

Error: Unexpected network error.

Checking the grub2 environment it does seem wierd that none of the networking variables are set they are completely blank.

grub_cpu=x86_64
grub_platform=efi
net_default_ip=(null)
net_default_mac=(null)
net_default_server=
package_version=2.02+dfsg1-12ubuntu2
prefix=
pxe_default_server=
root=

[Blake Rouse (blake-rouse)]

This bug blocks the progress of completion of bug: https://bugs.launchpad.net/maas/+bug/1820983

[Blake Rouse (blake-rouse)]

Seems that SUSE supports this in their grub because its well documented on their site:

https://en.opensuse.org/UEFI_HTTPBoot_Server_Setup

[Mathieu Trudel-Lapierre (cyphermox)]

As a first step I'd look into whether grub is able to correctly get an IP from DHCP. I wonder if the client class string confuses it.

Could you please take packet captures of the boot attempt? We could then have a look at the whole thing and compare with grub code to see what happens?

[Blake Rouse (blake-rouse)]

Attached is the packet capture.

[Mathieu Trudel-Lapierre (cyphermox)]

After additional testing with hardware (instead of in VMs), it looks like things are working fine on Lenovo hardware.

I managed to get access to a Lenovo SR670 server. I deployed it from an existing MAAS, along with a second server (some random node we'll call http-server) to be an HTTP server, since I couldn't update the existing MAAS server to the right commits.

So; http-server was a deployed node from MAAS, on which I simply installed 'apache2'. On that server, I removed index.html from /var/www/html and instead put in bootx64.efi, grubx64.efi, a grub directory with a custom grub.cfg file, and a directory structure copied from MAAS to have a kernel and initrd available. No further actions are required on http-server.

The existing MAAS server was slightly modified to point to the HTTP server: edited /var/lib/maas/dhcpd.conf to add the following snippet (same as in description with the URI updated for http-server):

"""
} elsif option arch = 00:0f {
    # uefi_amd64_http
    filename "http://http-server:80/bootx64.efi";
    option vendor-class-identifier "HTTPClient";
} elsif option arch = 00:10 {
    # uefi_amd64_http
    filename "http://http-server:80/bootx64.efi";
    option vendor-class-identifier "HTTPClient";
"""

Next; the Lenovo SR670 was configured to boot from the network using UEFI HTTP Boot (see https://lenovopress.com/lp0736.pdf); and restarted with a One-time Boot Option; where HTTP Boot was selected.

On the BMC for the Lenovo SR670; one can watch the system boot, request files from HTTP, and potentially show a grub menu or otherwise follow the instructions for booting listed in grub.cfg on http-server.

Watching on http-server, one can see in the apache access.log for requests from the Lenovo SR670:

10.193.36.45 - - [18/Apr/2019:16:10:36 +0000] "HEAD /bootx64.efi HTTP/1.1" 200 211 "-" "UefiHttpBoot/1.0"
10.193.36.45 - - [18/Apr/2019:16:10:36 +0000] "GET /bootx64.efi HTTP/1.1" 200 1335027 "-" "UefiHttpBoot/1.0"
10.193.36.45 - - [18/Apr/2019:16:10:37 +0000] "GET //grubx64.efi HTTP/1.1" 200 1164875 "-" "UefiHttpBoot/1.0"
10.193.36.45 - - [18/Apr/2019:16:10:37 +0000] "GET /grub/x86_64-efi/command.lst HTTP/1.1" 404 464 "-" "GRUB 2.02-2ubuntu8.13"
10.193.36.45 - - [18/Apr/2019:16:10:37 +0000] "GET /grub/x86_64-efi/fs.lst HTTP/1.1" 404 459 "-" "GRUB 2.02-2ubuntu8.13"
10.193.36.45 - - [18/Apr/2019:16:10:37 +0000] "GET /grub/x86_64-efi/crypto.lst HTTP/1.1" 404 463 "-" "GRUB 2.02-2ubuntu8.13"
10.193.36.45 - - [18/Apr/2019:16:10:37 +0000] "GET /grub/x86_64-efi/terminal.lst HTTP/1.1" 404 465 "-" "GRUB 2.02-2ubuntu8.13"
10.193.36.45 - - [18/Apr/2019:16:10:37 +0000] "GET /grub/grub.cfg HTTP/1.1" 200 338 "-" "GRUB 2.02-2ubuntu8.13"
10.193.36.45 - - [18/Apr/2019:16:10:37 +0000] "GET /grub/grub.cfg-default-amd64 HTTP/1.1" 200 278 "-" "GRUB 2.02-2ubuntu8.13"

This makes me conclude that grub2 does what it is supposed to, and thus any issues might be due to limitations in the EDK2 implementation of HTTP Boot -- possibly some registers or EFI variables are not being updated appropriately. Reassigning to edk2.