The proposed grub package in Trusty / Bionic overrides the boot order on a MaaS deployed system

[Po-Hsu Lin]

Similar to bug 1642298, bug 1750732 and bug 1788539

This issue was only spotted on ThunderX Cavium ARM64 system and an AMD Naples amd64 server in this SRU cycle.

Steps:
1. Deploy the ThunderX system with Trusty + T-LTS 4.4 kernel (It works fine with just Trusty, however this was tested on the AMD Naples, not ThunderX since it does not support 3.13 kernel)
2. Check the efibootmgr output
3. Enable proposed, install grub-common
4. Check efibootmgr again

Result:
In step 2, the boot order is correct:
ubuntu@wright-kernel:~$ sudo efibootmgr
BootCurrent: 0002
Timeout: 10 seconds
BootOrder: 0002,0000,0003,0004,0001
Boot0000* ubuntu
Boot0001 UEFI: Built-in EFI Shell
Boot0002* UEFI: IP4 Ethernet vNIC Controller 62:54:CA
Boot0003* UEFI: IP4 Ethernet vNIC Controller 62:54:CB
Boot0004* UEFI: IP4 Ethernet vNIC Controller 62:54:CC

But in step 4, the boot order will be overridden:
ubuntu@wright-kernel:~$ sudo efibootmgr
BootCurrent: 0002
Timeout: 10 seconds
BootOrder: 0000,0002,0003,0004,0001
Boot0000* ubuntu
Boot0001 UEFI: Built-in EFI Shell
Boot0002* UEFI: IP4 Ethernet vNIC Controller 62:54:CA
Boot0003* UEFI: IP4 Ethernet vNIC Controller 62:54:CB
Boot0004* UEFI: IP4 Ethernet vNIC Controller 62:54:CC

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: grub-common 2.02~beta2-9ubuntu1.15
ProcVersionSignature: User Name 4.4.0-134.160~14.04.1-generic 4.4.140
Uname: Linux 4.4.0-134-generic aarch64
ApportVersion: 2.14.1-0ubuntu3.29
Architecture: arm64
Date: Tue Sep 4 07:45:19 2018
SourcePackage: grub2
UpgradeStatus: No upgrade log present (probably fresh install)

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1790586

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Po-Hsu Lin (cypressyew)]

More test on the AMD Naples amd64 server "lodygin" shows:
1. This issue DOES NOT exist with this node deployed with Trusty.
2. This issue DOES exist with this node deployed with Trusty + T-LTS 4.4 kernel.
3. This issue DOES NOT exist with Xenial (no grub package in -proposed)
4. This issue DOES NOT exist with Xenial + X-HWE 4.15 kernel (no grub package in -proposed)
5. This issue DOES exist with this node deployed with Bionic.

More test on the ThunderX arm64 server "wright" shows:
1. (This node cannot be tested with Trusty)
2. This issue DOES exist with this node deployed with Trusty + X-HWE kernel.
3. This issue DOES NOT exist with Xenial (no grub package in -proposed though)
4. This issue DOES NOT exist with Xenial + X-HWE 4.15 kernel (no grub package in -proposed)
5. This issue DOES NOT exist with this node deployed with Bionic.

[dann frazier (dannf)]

In theory, you should *never* have an "ubuntu" entry in efibootmgr when MAAS-deployed. I'm guessing this is a dup of the following bug, for which a fix has not been backported to trusty:
  https://bugs.launchpad.net/maas/+bug/1642298

[Po-Hsu Lin (cypressyew)]

From this point I don't think this is a dup as I can see the "ubuntu" entry for both of the AMD Naples amd64 server and the TunderX arm64 server, with Bionic:

efibootmgr output after they have been deployed:
ubuntu@wright-kernel:~$ sudo efibootmgr
BootCurrent: 0002
Timeout: 10 seconds
BootOrder: 0002,0003,0004,0000,0001
Boot0000* ubuntu
Boot0001 UEFI: Built-in EFI Shell
Boot0002* UEFI: IP4 Ethernet vNIC Controller 62:54:CA
Boot0003* UEFI: IP4 Ethernet vNIC Controller 62:54:CB
Boot0004* UEFI: IP4 Ethernet vNIC Controller 62:54:CC

ubuntu@lodygin:~$ sudo efibootmgr
BootCurrent: 0002
Timeout: 1 seconds
BootOrder: 0002,0000,0003,0004
Boot0000* ubuntu
Boot0002* UEFI: IP4 Intel(R) Gigabit CT Desktop Adapter
Boot0003* UEFI: Built-in EFI Shell
Boot0004* Hard Drive

The only difference is that the grub package from proposed will change the boot order only on the AMD server:

ubuntu@lodygin:~$ sudo efibootmgr
BootCurrent: 0002
Timeout: 1 seconds
BootOrder: 0000,0002,0003,0004
Boot0000* ubuntu
Boot0002* UEFI: IP4 Intel(R) Gigabit CT Desktop Adapter
Boot0003* UEFI: Built-in EFI Shell
Boot0004* Hard Drive

[Po-Hsu Lin (cypressyew)]

For the AMD node lodygin, I can see this issue again in this cycle with Bionic

ubuntu@lodygin:~$ dpkg -l | grep grub | awk '{print$2,$3}'
grub-common 2.02-2ubuntu8.4
grub-efi-amd64 2.02-2ubuntu8.4
grub-efi-amd64-bin 2.02-2ubuntu8.4
grub-efi-amd64-signed 1.93.5+2.02-2ubuntu8.4
grub-pc 2.02-2ubuntu8.4
grub-pc-bin 2.02-2ubuntu8.4
grub2-common 2.02-2ubuntu8.4

ubuntu@lodygin:~$ uname -a
Linux lodygin 4.15.0-35-generic #38-Ubuntu SMP Wed Sep 12 10:38:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[Mathieu Trudel-Lapierre (cyphermox)]

My guess is lodygin fails to boot from the network, and cycles all the way to booting from Hard drive (0004), in which case, since shim and everything really /are/ installed on disk, it gets run to recreate the missing "ubuntu" entry.

Assuming you can reproduce this issue easily, can you check if it can be reproduced after removing the 0004 boot entry?

sudo efibootmgr -b -B 0004

[Mathieu Trudel-Lapierre (cyphermox)]

Not a kernel bug.

[Po-Hsu Lin (cypressyew)]

Node lodygin is out-of-order now, need to check this later.

[Launchpad Janitor (janitor)]

[Expired for grub2 (Ubuntu) because there has been no activity for 60 days.]

[Po-Hsu Lin (cypressyew)]

Didn't see this anymore, and as lodygin is dead, I am closing this bug.