[REGRESSION] ppc64el grub in bionic may be auto-generating MAC address instead of using the assigned to the VM.

[Andres Rodriguez]

[Impact]
grub2 on ppc64el wrongfully configures two interfaces for the same card when running in qemu with a tftp boot. This causes net_default_mac to point to the wrong mac address, amongst other things.

[Test case]
(1) grub-mknetdir --net-directory=$dir
(2) qemu-system-ppc64le -device virtio-net-pci,netdev=mynet,mac=$mac -netdev user,id=mynet,tftp=$dir,bootfile=boot/grub/powerpc-ieee1275/core.elf

grub loads successfully via tftp.

Verify that
(1) net_ls_cards shows the card with the correct mac addr $mac
(2) net_ls_addr shows one interface with mac $mac
(3) echo $net_default_mac shows the mac $mac

[Regression potential]
grub now correctly stores the mac address in variables, meaning that code now might not fallback to a default.

Apart from that/On the other hand, the fix is a simple case of initializing variables, therefore that was possible before too, as it was basically random anyway.

[Original bug report]
In MAAS, we use grub-mkimage to create a bootloader for VM's to PXE boot of MAAS by a grub bootloader that we provide over the network. (we call this bootppc64.bin).

This bootloader is created with the following grub configuration:

   grub_config: |
      # MAAS GRUB2 pre-loader configuration file

      # Load based on MAC address first.
      configfile (pxe)/grub/grub.cfg-${net_default_mac}

      # Failed to load based on MAC address.
      # Load arm64 by default, UEFI only supported by 64-bit
      configfile (pxe)/grub/grub.cfg-default-arm64

When we use bionic bootloader, we see the following output in the console:

 "Booting under MAAS direction... [ grub.cfg-default-ppc 642B 100% 57.16B/s ]"

And in rackd.log we see:

2018-08-07 17:37:21 provisioningserver.rackdservices.tftp: [info] bootppc64.bin requested by 10.245.136.136
2018-08-07 17:37:24 provisioningserver.rackdservices.tftp: [info] /grub/grub.cfg-80:1c:b0:c0:7d:c5 requested by 10.245.136.136
2018-08-07 17:37:24 provisioningserver.rackdservices.tftp: [info] /grub/grub.cfg-default-ppc64el requested by 10.245.136.136
2018-08-07 17:37:24 provisioningserver.rackdservices.tftp: [info] ubuntu/ppc64el/ga-18.04/bionic/daily/boot-kernel requested by 10.245.136.136

This means that the VM attempted to PXE boot with MAC address "80:1c:b0:c0:7d:c5", but the VM itself has a different MAC address "52:54:00:7f:83:62"

    <interface type='network'>
      <mac address='52:54:00:7f:83:62'/>
      <source network='maas'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>

If we are to switch the bootloader to the one created with xenial as a base, we see correct behavior.

 "Booting under MAAS direction... [ grub.cfg-52:54:00:7f 640B 100% 0.62B/s ] "

And the result is that grub has used to correct MAC address for the PXE process.

2018-08-07 17:07:45 provisioningserver.rackdservices.tftp: [info] bootppc64.bin requested by 10.245.136.136
2018-08-07 17:07:46 provisioningserver.rackdservices.tftp: [info] /grub/grub.cfg-52:54:00:7f:83:62 requested by 10.245.136.136
2018-08-07 17:07:47 provisioningserver.rackdservices.tftp: [info] ubuntu/ppc64el/generic/bionic/daily/boot-kernel requested by 10.245.136.136
2018-08-07 17:08:17 provisioningserver.rackdservices.tftp: [info] ubuntu/ppc64el/generic/bionic/daily/boot-initrd requested by 10.245.136.136

It would seem as the firmware in bionic is automatically generating a new MAC address.

Bionic console log
-------------------------------------------------------------------------------
Using default console: /vdevice/vty@30000000

  Welcome to Open Firmware

  Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Trying to load: from: /pci@800000020000000/ethernet@4 ...
 Initializing NIC
  Reading MAC address from device: 52:54:00:7f:83:62
  Requesting information via DHCP: done
  Using IPv4 address: 10.245.136.136
  Requesting file "bootppc64.bin" via TFTP from 10.245.136.6
  Receiving data: 1665 KBytes
  TFTP: Received bootppc64.bin (1665 KBytes)
  Successfully loaded
Booting under MAAS direction... [ grub.cfg-default-ppc 642B 100% 57.16B/s ]
OF stdout device is: /vdevice/vty@30000000initrd 55.89MiB 100% 647.54KiB/s ]
Preparing to boot Linux version 4.15.0-30-generic (buildd@bos02-ppc64el-011) (gcc version 7.3.0 (Ubuntu 7
.3.0-16ubuntu3)) #32-Ubuntu SMP Thu Jul 26 17:43:11 UTC 2018 (Ubuntu 4.15.0-30.32-generic 4.15.18)
Detected machine type: 0000000000000101
command line: BOOT_IMAGE=ubuntu/ppc64el/ga-18.04/bionic/daily/boot-kernel nomodeset ro root=squash:http:/
/10.245.136.6:5248/images/ubuntu/ppc64el/ga-18.04/bionic/daily/squashfs ip=::::maas-enlist:BOOTIF ip6=off
 overlayroot=tmpfs overlayroot_cfgdisk=disabled cc:{datasource_list: [MAAS]}end_cc cloud-config-url=http:
//10-245-136-0--21.maas-internal:5248/MAAS/metadata/latest/enlist-preseed/?op=get_enlist_preseed apparmor
=0 log_host=10.245.136.6 log_port=514 BOOTIF=01-80:1c:b0:c0:7d:c5

Xenial console log
--------------------------------------------------------------------
Using default console: /vdevice/vty@30000000 [1032/2975]

  Welcome to Open Firmware

  Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Trying to load: from: /pci@800000020000000/ethernet@4 ...
 Initializing NIC
  Reading MAC address from device: 52:54:00:7f:83:62
  Requesting information via DHCP: done
  Using IPv4 address: 10.245.136.136
  Requesting file "bootppc64.bin" via TFTP from 10.245.136.6
  Receiving data: 1714 KBytes
  TFTP: Received bootppc64.bin (1714 KBytes)
  Successfully loaded
Booting under MAAS direction... [ grub.cfg-52:54:00:7f 640B 100% 0.62B/s ]
OF stdout device is: /vdevice/vty@30000000initrd 55.89MiB 100% 958.95KiB/s ]
Preparing to boot Linux version 4.15.0-30-generic (buildd@bos02-ppc64el-011) (gcc version 7.3.0 (Ubuntu $
.3.0-16ubuntu3)) #32-Ubuntu SMP Thu Jul 26 17:43:11 UTC 2018 (Ubuntu 4.15.0-30.32-generic 4.15.18)
Detected machine type: 0000000000000101
command line: BOOT_IMAGE=ubuntu/ppc64el/generic/bionic/daily/boot-kernel nomodeset ro root=squash:http:/$
10.245.136.6:5248/images/ubuntu/ppc64el/generic/bionic/daily/squashfs ip=::::fast-cow:BOOTIF ip6=off ove$
layroot=tmpfs overlayroot_cfgdisk=disabled cc:{datasource_list: [MAAS]}end_cc cloud-config-url=http://10$
245-136-0--21.maas-internal:5248/MAAS/metadata/latest/by-id/4ysp63/?op=get_preseed apparmor=0 log_host=1$
.245.136.6 log_port=514 rootdelay=60 BOOTIF=01-52:54:00:7f:83:62
Max number of cores passed to firmware: 2048 (NR_CPUS = 2048)

[Lee Trager (ltrager)]

I have reverted the OpenFirmware PPC64EL bootloader in lp:maas-images to Xenial while this is debugged.

[Julian Andres Klode (juliank)]

I ran qemu (in cosmic) with a cloud image and

 -device virtio-net-pci,netdev=mynet,mac=$mymac -netdev user,id=mynet

and loaded the ofnet module. I then listed the network cards and the mac address was correctly recognized.

I have not figured out how to PXE boot it yet, so net_default_mac was empty.

But this experience leads me to believe that there are multiple interfaces and it might be looking at the wrong one. There does not seem to be any code in grub to generate random mac addresses.

[Julian Andres Klode (juliank)]

Reproduced:

(1) grub-mknetdir --net-directory=$dir
(2) qemu-system-ppc64le -device virtio-net-pci,netdev=mynet,mac=$mac -netdev user,id=mynet,tftp=$dir,bootfile=boot/grub/powerpc-ieee1275/core.elf

grub loads successfully via tftp.

Observations:

(1) net_ls_cards shows "ofnet_net $mac", so correctly recognizes the mac address.
(2) net_default_interface = ofnet_net
(3) net_default_mac and net_ofnet_net_mac are different

[Julian Andres Klode (juliank)]

Running net_ls_addr shows two ofnet_net, one with the correct MAC and IP address, and one with the fake MAC and "Unknown address type 31"

[Julian Andres Klode (juliank)]

This is the culprit:

https://git.savannah.gnu.org/cgit/grub.git/commit/grub-core/net?id=9585647a25b65f98db6bd22c569b34795512f046

Removing the parser for bootpath gets rid of the broken additional interface. I expect that checking the return value of grub_ieee1275_get_property() when querying the mac also fixes it, and will investigate that shortly (as it seems the bootpath contains IP addresses, but not MAC addresses).

[Julian Andres Klode (juliank)]

It turned out to be uninitialized local variables. Patch sent upstream.

https://lists.gnu.org/archive/html/grub-devel/2018-08/msg00073.html

[Julian Andres Klode (juliank)]

Merge proposal for packaging branch:

https://code.launchpad.net/~juliank/grub/+git/ubuntu/+merge/353662

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02+dfsg1-5ubuntu1

---------------
grub2 (2.02+dfsg1-5ubuntu1) cosmic; urgency=medium

  [ Mathieu Trudel-Lapierre]
  * Merge against Debian unstable; remaining changes:
    - debian/control: Update Vcs fields for code location on Ubuntu.
    - debian/control: Breaks shim (<< 13).
    - Secure Boot support: use newer patchset from rhboot repo:
      - many linuxefi_* patches added and modified
      - dropped debian/patches/linuxefi_require_shim.patch
      - renamed: debian/patches/no_insmod_on_sb.patch ->
        debian/patches/linuxefi_no_insmod_on_sb.patch
    - debian/patches/install_signed.patch, grub-install-extra-removable.patch:
      - Make sure if we install shim; it should also be exported as the default
        bootloader to install later to a removable path, if we do.
      - Rework grub-install-extra-removable.patch to reverse its logic: in the
        default case, install the bootloader to /EFI/BOOT, unless we're trying
        to install on a removable device, or explicitly telling grub *not* to
        do it.
      - Move installing fb$arch.efi to --no-extra-removable; as we don't want
        fallback to be installed unless we're also installing to /EFI/BOOT.
        (LP: #1684341)
      - Install a BOOT.CSV for fallback to use.
      - Make sure postinst and templates know about the replacement of
        --force-extra-removable with --no-extra-removable.
    - debian/patches/add-an-auto-nvram-option-to-grub-install.patch: Add the
      --auto-nvram option to grub-install for auto-detecting NVRAM availability
      before attempting NVRAM updates.
    - debian/build-efi-images: provide a new grub EFI image which enforces that
      loaded kernels are signed for Secure Boot: build gsb$arch.efi; which is
      the same as grub$arch.efi minus the 'linux' module. Without fallback to
      'linux' for unsigned loading, this makes it effectively enforce having a
      signed kernel. (LP: #1401532)
    - Verify that the current and newer kernels are signed when grub is
      updated, to make sure people do not accidentally shutdown without a
      signed kernel.
    - debian/default/grub: replace GRUB_HIDDEN_* variables with the less
      confusing GRUB_TIMEOUT_STYLE=hidden. (LP: #1258597)
    - debian/patches/support_initrd-less_boot.patch: Added knobs to allow
      non-initrd boot config. (LP: #1640878)
    - Disable os-prober for ppc64el on the PowerNV platform, to reduce the
      number of entries/clutter from other OSes in Petitboot (LP: #1447500)
    - debian/patches/shorter_version_info.patch: Only show the upstream version
      in menu and console, and hide the package one in a package_version
      variable. (LP: #1723434)
    - debian/patches/skip_text_gfxpayload_where_not_supported.patch: Skip the
      'text' payload if it's not supported but present in gfxpayload, such as
      on EFI systems. (LP: #1711452)
    - debian/patches/bufio_sensible_block_sizes.patch: Don't use arbitrary file
      fizes as block sizes in bufio: this avoids potentially seeking back in
      the files unnecessarily, which may require re-open files that cannot be
      seeked into, such as via...

[Łukasz Zemczak (sil2100)]

Hello Andres, or anyone else affected,

Accepted grub2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02-2ubuntu8.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Julian Andres Klode (juliank)]

old package (2.02-2ubuntu8.3) is bad:

grub> net_ls_cards
ofnet_net a1:b2:c3:d4:e5:f6
grub> net_ls_addr
ofnet_net a1:b2:c3:d4:e5:f6 10.0.2.15
ofnet_net 1f:ff:c1:20:1d:c5 Unknown address type 31
grub> echo $net_default_mac
1f:ff:c1:20:1d:c5

new package (2.02-2ubuntu8.3) is good:
grub> net_ls_cards
ofnet_net a1:b2:c3:d4:e5:f6
grub> net_ls_addr
ofnet_net a1:b2:c3:d4:e5:f6 10.0.2.15
grub> echo $net_default_mac
a1:b2:c3:d4:e5:f6

[Julian Andres Klode (juliank)]

oops, I obviously meant:

new package (2.02-2ubuntu8.4) is good:
                           ^

not 8.3

[Nicholas Hinds (hindsn)]

Installing the grub packages for this bug from the bionic-proposed repository removes grub-efi-amd64-signed and shim-signed. The removal of shim-signed seems to stop DKMS modules from being signed.

Start-Date: 2018-08-29 12:49:47
Commandline: apt dist-upgrade
Requested-By: nh (1000)
Upgrade: grub-common:amd64 (2.02-2ubuntu8.1, 2.02-2ubuntu8.4), grub2-common:amd64 (2.02-2ubuntu8.1, 2.02-2ubuntu8.4), grub-efi-amd64-bin:amd64 (2.02-2ubuntu8.1, 2.02-2ubuntu8.4), grub-efi-amd64:amd64 (2.02-2ubuntu8.1, 2.02-2ubuntu8.4)
Remove: shim-signed:amd64 (1.34.9.2+13-0ubuntu2), grub-efi-amd64-signed:amd64 (1.93.1+2.02-2ubuntu8.1)
End-Date: 2018-08-29 12:50:23

Downgrading to version 8.3 allows shim-signed to be reinstalled, and DKMS modules have signatures again.

> sudo apt install grub-common=2.02-2ubuntu8.3 grub2-common=2.02-2ubuntu8.3 grub-efi-amd64-bin=2.02-2ubuntu8.3 grub-efi-amd64=2.02-2ubuntu8.3 shim-signed=1.34.9.2+13-0ubuntu2 grub-efi-amd64-signed=1.93.4+2.02-2ubuntu8.3

[Julian Andres Klode (juliank)]

Yes, there needs to be a rebuild of grub-signed too before it can be released.

[Łukasz Zemczak (sil2100)]

Hello Andres, or anyone else affected,

Accepted grub2-signed into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.93.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Julian Andres Klode (juliank)]

Marking back as verified, as that upload of -signed was just for the signatures, thus does not invalidate the testing done with the unsigned one before.

When this will be released, we need to be careful to release both at the same time :)

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for grub2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.93.5

---------------
grub2-signed (1.93.5) bionic; urgency=medium

  * Rebuild against grub2 2.02-2ubuntu8.4 (LP: #1785859)

 -- Julian Andres Klode <email address hidden> Thu, 30 Aug 2018 16:22:15 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02-2ubuntu8.4

---------------
grub2 (2.02-2ubuntu8.4) bionic; urgency=medium

  * debian/patches/ofnet-init-structs-in-bootpath-parser.patch: initialize
    structs in bootpath parser. Fixes netboot issues on ppc64el. (LP: #1785859)

 -- Julian Andres Klode <email address hidden> Thu, 23 Aug 2018 21:29:46 +0200