[OVN][GRUB] Grub bootloader doesn't netboot with OVN DHCP

[Julia Kreger]

In attempting to get OVN networking working, we appear to have an issue where our network boot operations fail:

Specifically, what we're seeing is:

* DHCP response is issued:
  - Opt 53 - ACK
  - Opt 67 - bootfile name (bootx64.efi)
  - Opt 253 Private
  - Opt 121 - classless static route -> metadata IP to 10.1.0.2 (The OVN Metadata agent)
  - Opt 6 - DNS 127.0.0.1 (gah!)
  - Opt 51 Lease time - 12 hours
  - Opt 26 - mtu 1430 bytes (where does it get this?!?!)
  - Opt 1 - subnet mask 255.255.255.192
  - Opt 210 - PXE path prefix to the appropriate tftpboot folder path
  - Opt 3 - router 10.1.0.1
  - Opt 54 - dhcp server identifier 10.1.0.1
  - Opt 66 - dhcp server name: 10.208.22.67
  - Opt 150 - DHCP server address 10.208.225.67

* TFTP transfer starts
- VM grabs bootx64.efi, completes
- VM grabs grubx64.efi, completes

And then promptly starts arping to the wire "Who has 10.208.225.67? Tell 10.1.0.34"

As if it lost through tthe configuration load what it's routing table is.

And we can confirm this by passing net_ls_routes:

grub> net_ls_routes
efinet0:local 10.0.0.0/8 efinet0
grub>

Which makes this seem like a grub bug, but given the fact this worked with OVS + dnsmasq dhcp, we need to understand the root cause.

[Julia Kreger (juliaashleykreger)]

Captured via tcpdump:

19:52:40.983194 IP (tos 0xc0, ttl 64, id 30528, offset 0, flags [none], proto UDP (17), length 385)
    10.1.0.2.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 357, xid 0xfc64e08, Flags [Broadcast]
   Your-IP 10.1.0.52
   Server-IP 213.32.74.11
   Client-Ethernet-Address 52:54:00:67:75:9b
   Vendor-rfc1048 Extensions
     Magic Cookie 0x63825363
     DHCP-Message (53), length 1: Offer
     Server-ID (54), length 4: 10.1.0.2
     Lease-Time (51), length 4: 86400
     RN (58), length 4: 43200
     RB (59), length 4: 75600
     Subnet-Mask (1), length 4: 255.255.255.192
     BR (28), length 4: 10.1.0.63
     Domain-Name-Server (6), length 4: 10.1.0.2
     Domain-Name (15), length 14: "openstacklocal"
     Hostname (12), length 14: "host-10-1-0-52"
     BF (67), length 12: "bootx64.efi^@"
     TFTP (66), length 13: "213.32.74.11^@"
     Default-Gateway (3), length 4: 10.1.0.1
     MTU (26), length 2: 1380

Major differences appear to be option 210 (path), 121 (static route), and of course this response worked just fine.

[Julia Kreger (juliaashleykreger)]

I've posted a patch to test if it is option 210, in large part because it seemed reasonable cleanup.

If it is, indeed, option 121, then we will need to consult with neutron/OVN.

[Julia Kreger (juliaashleykreger)]

Test reveals it seems to be the classless route. :\

[Julia Kreger (juliaashleykreger)]

Well, no specific metadata route, and no option 210, and we still have the same behavior with grub.

I'll reach out to some grub developers next week and see if they are aware of any such behavior.

a20:50:41.821708 IP (tos 0x0, ttl 64, id 49160, offset 0, flags [none], proto UDP (17), length 353)
    10.1.0.1.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 325, xid 0xd3370e75, Flags [Broadcast]
   Your-IP 10.1.0.30
   Server-IP controller
   Client-Ethernet-Address 52:54:00:fd:56:02 (oui Unknown)
   Vendor-rfc1048 Extensions
     Magic Cookie 0x63825363
     DHCP-Message (53), length 1: Offer
     BF (67), length 11: "bootx64.efi"
     Unknown (253), length 4: 181485885
     Domain-Name-Server (6), length 4: localhost
     Lease-Time (51), length 4: 43200
     MTU (26), length 2: 1430
     Subnet-Mask (1), length 4: 255.255.255.192
     Default-Gateway (3), length 4: 10.1.0.1
     Server-ID (54), length 4: 10.1.0.1
     TFTP (66), length 12: "10.209.65.61"
     TFTP-Server-Address (150), length 4: controller

[Julia Kreger (juliaashleykreger)]

I sought insight from Red Hat's grub maintainers, and they suggested building a EFI grub payload which injects a config to obtain some debugging.

That, itself didn't quite work so well, but I did add some commands to gather some additional data:

Specifically:

set=debug=efinet,dhcp
net_ls_routes
net_ls_addr
net_dhcp
net_ls_addr
net_ls_routes
normal

which got me:

"""
Fetching Netboot Image
efinet0:local 10.0.0.0/8 efinet0

efinet0 52:54:00:fd:56:02 10.1.0.58

efinet0:dhcp 52:54:00:fd:56:02 10.1.0.58

efinet0 52:54:00:fd:56:02 10.1.0.58

efinet0:dhcp:default 0.0.0.0/0 gw 10.1.0.1

efinet0:dhcp:local 10.1.0.0/26 efinet0:dhcp

efinet0:local 10.0.0.0/8 efinet0

GNU GRUB version 2.06

   Minimal BASH-like line editing is supported. For the first word, TAB

   lists possible command completions. Anywhere else TAB lists possible

   device or file completions.

grub>"""

The interesting aspect is even after re-dhcping, we observe the same behavior, the image goes on trying to arp on 10.0.0.0/8 even though the dhcp payload indicates a smaller network.

Additionally, I pulled the fedora 38 binaries for shim/grub and loaded them and encountered the same behavior in terms of grub is now just sitting trying to arp for the tftp server's address as if it is locally attached instead of through the default gateway/router.

Looking at the grub include/grub/net.h file, the only options from the response payload which are used with dnsmasq, but which are not present with OVN is GRUB_NET_BOOTP_HOSTNAME (option 12) and GRUB_NET_BOOTP_DOMAIN (option 15), yet they are not really used either. The other major difference in used options appears to be the GRUB_NET_BOOTP_DNS option, which does get a bogus (127.0.0.1) response from OVN.

This makes no sense.

[Julia Kreger (juliaashleykreger)]

So it occured to me that the tftp_server address was different between the CI runs. With the OVN jobs, we generally seemed to end up with a HOST_IP parameter which was not in 10.0.0.0/8.

So I changed the IP out, and magically it worked. It *could* be that there has always sort of been a failure case possible to hit with grub, that we just happened to stumble upon with OVN.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ironic/+/893981

[Julia Kreger (juliaashleykreger)]

Well, further I guess.

Pinned the server IP address, and we complete the binary bytes transfer, but then break down *afterwards*

Sep 06 22:10:19 np0035178065 in.tftpd[123739]: remap: input: bootx64.efi
Sep 06 22:10:19 np0035178065 in.tftpd[123739]: remap: rule 3: rewrite: /opt/stack/data/ironic/tftpboot/bootx64.efi
Sep 06 22:10:19 np0035178065 in.tftpd[123739]: remap: rule 3: exit
Sep 06 22:10:19 np0035178065 in.tftpd[123739]: RRQ from 10.1.0.58 filename bootx64.efi remapped to /opt/stack/data/ironic/tftpboot/bootx64.efi
Sep 06 22:10:19 np0035178065 in.tftpd[123739]: tftp: client does not accept options
Sep 06 22:10:19 np0035178065 in.tftpd[123740]: remap: input: bootx64.efi
Sep 06 22:10:19 np0035178065 in.tftpd[123740]: remap: rule 3: rewrite: /opt/stack/data/ironic/tftpboot/bootx64.efi
Sep 06 22:10:19 np0035178065 in.tftpd[123740]: remap: rule 3: exit
Sep 06 22:10:19 np0035178065 in.tftpd[123740]: RRQ from 10.1.0.58 filename bootx64.efi remapped to /opt/stack/data/ironic/tftpboot/bootx64.efi
Sep 06 22:10:20 np0035178065 in.tftpd[123741]: remap: input: grubx64.efi
Sep 06 22:10:20 np0035178065 in.tftpd[123741]: remap: rule 3: rewrite: /opt/stack/data/ironic/tftpboot/grubx64.efi
Sep 06 22:10:20 np0035178065 in.tftpd[123741]: remap: rule 3: exit
Sep 06 22:10:20 np0035178065 in.tftpd[123741]: RRQ from 10.1.0.58 filename grubx64.efi remapped to /opt/stack/data/ironic/tftpboot/grubx64.efi

But if we look at the console log:

[=3h
>>Start PXE over IPv4.
  Station IP address is 10.1.0.58

  Server IP address is 172.24.5.1
  NBP filename is bootx64.efi
  NBP filesize is 960472 Bytes
 Downloading NBP file...

  NBP file downloaded successfully.
BdsDxe: loading Boot0001 "UEFI PXEv4 (MAC:52540044EED3)" from PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(52540044EED3,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)
BdsDxe: starting Boot0001 "UEFI PXEv4 (MAC:52540044EED3)" from PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(52540044EED3,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)
Fetching Netboot Image
GNU GRUB version 2.06

   Minimal BASH-like line editing is supported. For the first word, TAB

   lists possible command completions. Anywhere else TAB lists possible

   device or file completions.

grub>

Nothing in the firewall log indicating dropped packets, I'll likely need to get on this node and tcpdump it directly.

Interestingly enough, this worked in other local testing, but it may have been with my modified re-dhcp grub2 efi image.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Julia Kreger <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/ironic/+/893981

[Alan Baghumian (alanbach)]

Hi Julia! I was working on a project this weekend which involves PXE booting VM(s) using MAAS inside an OVN subnet and encounter a very similar behavior.

- The VM gets an IP address from MAAS and switches to PXE
- The VM downloads bootx64.efi and grubx64.efi
- The process fails with PXE-E99: Unexpected Network Error
- Packet captures show "Who has <IP>? Tell <IP> messages

To workaround this issue, I ended up configuring the VM(s) to use legacy boot and everything works after that.

I'll try to re-capture some pcap files and attach them here.

[Julia Kreger (juliaashleykreger)]

Greetings Alan!

We're basically expecting legacy boot to disappear given the increasing shift of CPUs which don't natively support it.

That being said, in your test was the "Who has <IP>" portion of your packet captures, was the IP grub thinks the DHCP server is?

AIUI, there is no actual DHCP IP server bind with OVN, the packets just get injected, and I think that is ultimately what breaks the interaction if grub is trying to ARP the dhcp server.

[Alan Baghumian (alanbach)]

PCAP from EFI PXE session

[Alan Baghumian (alanbach)]

PCAP from Legacy PXE session

[Alan Baghumian (alanbach)]

I attempted a boot using EFI, captured a PCAP, then switched the instance to legacy boot and captured a PCAP again. If you are curious check these.