grubnetx64.efi tftp client does not work over ipv6

[Steve Langasek]

[Impact]
Trying to PXE boot (with UEFI) over IPv6 with grub. This is especially relevant for MAAS users in IPv6 setups.

[Test cases]

[IPv6 PXE boot]
0) Setup PXE boot infrastructure (https://wiki.ubuntu.com/UEFI/SecureBoot-PXE-IPv6, also https://github.com/openSUSE/kiwi/wiki/Setup-PXE-boot-with-EFI-Using-GRUB2 contains relevant information)
1) Attempt to PXE boot (with UEFI) over IPv6 a system.

[IPv4 PXE boot]
0) Setup PXE boot infrastructure;
Required DHCP config:
 - filename should specify the path to the efi binary to use to boot, or to pxelinux in a non-EFI case.
 - next-server should be set to the TFTP server.
1) Attempt to PXE boot (with and without UEFI) with IPv4.

[IPv4 PXE boot with HTTPClient]
0) Setup PXE boot infrastructure;
Required DHCP config:
 - set vencor class to HTTPClient.
 - configure filename to be the url to the pxelinux (non-UEFI) or EFI binary (UEFI)
 - next-server should be set to the TFTP server.
1) Attempt to PXE boot (with and without UEFI) with IPv4.

[MaaS deployment in IPv4 network]
1) deploy a system using MaaS in an IPv4-only network.

[MaaS deployment in IPv6 network]
1) deploy a system using MaaS in an IPv6-only network.

[MaaS deployment in a mixed network]
1) deploy a system using MaaS on a network with both IPv4 and IPv6.

Testers can use grubnetx64.efi.signed from http://archive.ubuntu.com/ubuntu/dists/xenial-proposed/main/uefi/grub2-amd64/current/ once the package is available in proposed.

[Regression potential]
Possible regressions include any issues in routing IPv4 or IPv6 and/or retrieving files over PXE/tftp via grub. Ping is a good way to validate that routing is being done correctly, so is actually booting using the build grubnetx64.efi.

---

Testing using the pre-built grubnetx64.efi, I am not able to use the tftp client support within grub2 to load configs (/kernels/initrds) over the network. This works fine if using IPv4.

grubnetx64.efi is being loaded over the network (via shim no less), so the firmware's network stack is definitely up and working. But when grub tries to load grub.cfg via the default path, it fails with:

  error: couldn't send network packet.

This message is repeated, approximately once every three seconds. It looks to be an infinite loop; at least, the message is repeated more than 100 times. But sometimes, when I've not been paying close attention to the boot, I get a grub shell instead. In that case, the grub shell shows:

grub> echo $root
tftp,0.5.0.24
grub> set
[...]
net_default_server=
net_efinet2_boot_file=8:23f::2]/bootx64.efi
net_efinet2_ip=0.0.0.0
net_efinet2_mac=02:3f:00:00:00:00
[...]
prefix=(tftp,0.5.0.24)/grub
pxe_default_server=
root=tftp,0.5.0.24
[...]
grub>

The actual server IP is 2001:1938:23f::2.

I've booted a locally-generated (self-signed) grubnetx64.efi with grub-bootstrap.cfg modified to call 'set' first, and I get identical output for all of the network-related variables.

[Steve Langasek (vorlon)]

Details on how to configure a DHCPv6 server in order to reproduce this can be found here:
  https://wiki.ubuntu.com/UEFI/SecureBoot-PXE-IPv6#DHCPv6_.28isc-dhcp-server.29

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

This bug is still present in grub2 2.02~beta2-21 in vivid. The output is no longer the same - the "Error: couldn't send network packet" message is no longer shown - but the boot eventually drops to a grub shell and the environment shows the same kind of corruption as before, with a broken IPv4 address of "0.5.0.24" everywhere.

[Mike Pontillo (mpontillo)]

I'm curious if there is any change in behavior when using a hostname that resolves to a AAAA record instead of an IP address. Browsing the grub source code, it looks like if it has an IP address, it assumes IPv4 and generates that (tftp,x.x.x.x) string you saw.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02~beta2-36ubuntu10

---------------
grub2 (2.02~beta2-36ubuntu10) yakkety; urgency=medium

  * debian/patches/ip6_send_router_solicitation_7c4b6b7b.patch: handle long
    RA intervals by explicitly sending a SOLICIT.
  * debian/patches/ip6_fix_routing_eb9f401f.patch: fix IPv6 routing; we should
    be able to talk to things outside of link-local addresses; to do this,
    allow specifying a gateway and interface. (LP: #1229458)

 -- Mathieu Trudel-Lapierre <email address hidden> Wed, 27 Jul 2016 16:02:18 -0400

[Brian Murray (brian-murray)]

Hello Steve, or anyone else affected,

Accepted grub2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02~beta2-36ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Steve, or anyone else affected,

Accepted grub2-signed into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.66.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mike Pontillo (mpontillo)]

Adding the MAAS project so that we have a vehicle to confirm the fix in MAAS.

[LaMont Jones (lamont)]

With this dhcpv6 and the bootloaders from yakkety (as of yesterday's date), we get http://people.canonical.com/~lamont/1229458-screenshot.jpg (I'll attach that momentarily). It appears that bootx64.efi finished loading grub, released the IP, got the reply for the release, and launched grubx64.efi -- no evidence of dhcp solicit happening from grub.

[Steve Langasek (vorlon)]

> It appears that bootx64.efi finished loading grub, released the
> IP, got the reply for the release, and launched grubx64.efi

That sounds buggy to me, why should shim release the IP instead of leaving it provisioned in firmware? I'm sure we do still need grub fixed to correctly dhcp in the case where no IP is configured, but it's surely less than ideal for netboot to require up to 4 DHCP round trips (firmware, shim, grub, linux) if the results could be stashed in firmware.

[Mathieu Trudel-Lapierre (cyphermox)]

FWIW, I'm not sure what was releasing the IP; I only noticed that it was being released. I've done more work on this and while I didn't notice this behavior (didn't look, to be honest...) it appears that grub2 is behaving correctly in terms of configuring the efinet device for IPv6, and apparently correctly requests files via TFTP.

I've seen timeouts retrieving the kernel image, but there *were* RRQs received on the TFTP server for the file, it just seems it never got a response. I'm not sure yet why this happened, I suspect it may be due to a poor tftp configuration for my tests.

[Mathieu Trudel-Lapierre (cyphermox)]

Fixed in yakkety:

grub2 (2.02~beta2-36ubuntu11) yakkety; urgency=medium

  * debian/control: don't force building with GCC 5 when 6 is now the default.
  * support_module_without_symbol_table.patch: fix checks for modules without
    a symbol table to be allowed, since binutils' 'strip --stripunneeded' no
    longer leaves a symbols section around after stripping.
  * Fix support for IPv6 PXE booting under UEFI:
    - grub_add_grub_env_set_net_property.patch: add grub_env_set_net_property.
    - misc-fix-invalid-char-strtol.patch: fix strto*l methods invalid chars.
    - net_read_bracketed_ipv6_addr.patch: read bracketed IPv6 addresses.
    - bootp_new_net_bootp6_command.patch: add new bootp6 commands.
    - efinet_uefi_ipv6_pxe_support.patch: teach efinet to allow bootp6.
    - bootp_process_dhcpack_http_boot.patch: process DHCPACK, support HTTP.
    - efinet_set_network_from_uefi_devpath.patch: configure network from the
      devpath provided by the UEFI firmware.
    - efinet_set_dns_from_uefi_proto.patch: set DNS nameservers and search
      domains from the UEFI protocol.

[Ubuntu Foundations Team Bug Bot (crichton)]

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of grub2-signed from xenial-proposed was performed and bug 1627584 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed" tag from this bug report and add the tag "bot-stop-nagging" to bug 1627584 (not this bug). Thanks!

[Andy Whitcroft (apw)]

@cyphermox -- having reviewed the SRU request in the queue, overall it is looking reasonable as it is. There is some concern that there is some potential for functionality changes for IPv4 clients. Also it would be good to understand better how the HTTPClient vendor extension might affect IPv4. Could we update the SRU template to include explicit test cases IPv4 (including something which will tickle the EFI IPv4 changes) to confirm we are not regressing there once this hits -proposed.

[Mathieu Trudel-Lapierre (cyphermox)]

This is a functionality change inasmuch as it gives admins the possibility to use the same method for booting IPv4 and IPv6 using a URL for HTTPClient rather than relying on the next-server/tftp for IPv4 only (and thus a path) and either URL or path for IPv6 -- it's cleanup that goes with the rest of the patchset for IPv6 support.

I've updated the description for the test cases.

[Andy Whitcroft (apw)]

Hello Steve, or anyone else affected,

Accepted grub2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02~beta2-36ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Andy Whitcroft (apw)]

Hello Steve, or anyone else affected,

Accepted grub2-signed into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.66.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Steve, or anyone else affected,

Accepted grub2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02~beta2-36ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Steve, or anyone else affected,

Accepted grub2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02~beta2-36ubuntu3.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

An upload of grub2-signed to xenial-proposed has been rejected from the upload queue for the following reason: "wrong build-depends".

[Steve Langasek (vorlon)]

Hello Steve, or anyone else affected,

Accepted grub2-signed into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.66.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[LaMont Jones (lamont)]

Verified that both IPv4 pxeboot and IPv6 uefi boot cleanly into MAAS.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02~beta2-36ubuntu3.6

---------------
grub2 (2.02~beta2-36ubuntu3.6) xenial; urgency=medium

  * Fix support for IPv6 PXE booting under UEFI: (LP: #1229458)
    - grub_add_grub_env_set_net_property.patch: add grub_env_set_net_property.
    - misc-fix-invalid-char-strtol.patch: fix strto*l methods invalid chars.
    - net_read_bracketed_ipv6_addr.patch: read bracketed IPv6 addresses.
    - bootp_new_net_bootp6_command.patch: add new bootp6 commands.
    - efinet_uefi_ipv6_pxe_support.patch: teach efinet to allow bootp6.
    - bootp_process_dhcpack_http_boot.patch: process DHCPACK, support HTTP.
    - efinet_set_network_from_uefi_devpath.patch: configure network from the
      devpath provided by the UEFI firmware.
    - efinet_set_dns_from_uefi_proto.patch: set DNS nameservers and search
      domains from the UEFI protocol.
  * Fix booting on Hyper-V gen 2 VMs due to the lack of PIT there; we can deal
    with this by using other timers when PIT aren't available. (LP: #1519836)
    - debian/patches/git_tsc_use_alt_delay_sources_d43a5ee6.patch
    - debian/patches/git_split_pmtimer_wait_tsc_d9a3bfea.patch
    - debian/patches/git_fix_tsc_calibration_pit_a03c1034.patch

grub2 (2.02~beta2-36ubuntu3.3) xenial; urgency=medium

  * debian/patches/ip6_send_router_solicitation_7c4b6b7b.patch: handle long
    RA intervals by explicitly sending a SOLICIT.
  * debian/patches/ip6_fix_routing_eb9f401f.patch: fix IPv6 routing; we should
    be able to talk to things outside of link-local addresses; to do this,
    allow specifying a gateway and interface. (LP: #1229458)

 -- Mathieu Trudel-Lapierre <email address hidden> Thu, 15 Sep 2016 13:56:55 -0400

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for grub2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.66.6

---------------
grub2-signed (1.66.6) xenial; urgency=medium

  * Rebuild against grub2 2.02~beta2-36ubuntu3.6. (LP: #1229458, #1519836)

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 19 Dec 2016 21:12:45 -0500