grub package will change the boot order for MaaS deployed system

[Po-Hsu Lin]

Similar to bug 1642298, this issue strikes back (looks like it's only affecting our node "lodygin - AMD Speedway Naples 2U" in the test pool)

Steps:
 1. Deploy a system with Xenial
 2. Check the boot order with `efibootmgr`
 3. Upgrade grub-efi-amd64 from proposed
 4. Reboot
 5. Check the `efibootmgr` again

Result:
 * The boot order will be overrided to boot from disk first, MaaS needs it to boot with PXE first.

Package version:
grub-common 2.02~beta2-36ubuntu3.17
grub-efi-amd64 2.02~beta2-36ubuntu3.17
grub-efi-amd64-bin 2.02~beta2-36ubuntu3.17
grub-efi-amd64-signed 1.66.17+2.02~beta2-36ubuntu3.17
grub-pc 2.02~beta2-36ubuntu3.16
grub-pc-bin 2.02~beta2-36ubuntu3.17
grub2-common 2.02~beta2-36ubuntu3.17

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: grub-efi-amd64 2.02~beta2-36ubuntu3.17
ProcVersionSignature: User Name 4.4.0-112.135-generic 4.4.98
Uname: Linux 4.4.0-112-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.15
Architecture: amd64
Date: Wed Feb 21 06:09:29 2018
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: grub2
UpgradeStatus: No upgrade log present (probably fresh install)

[dann frazier (dannf)]

The change in GRUB for bug 1642298 only adds a hook to disable updating NVRAM - it doesn't change any defaults. The code to do that is in curtin and/or MAAS. What versions of curtin/MAAS are in use here?

[Andres Rodriguez (andreserl)]

MAAS sends a debconf selection to curtin, so that curtin tells grub to not update the nvram. The information that's required is:

1. The curtin config sent to curtin (maas <user> machine get-curtin-config <system_id>)
2. The installation log (you can grab both the UI/API)
3. The *actual* debconf selection set on the system (i guess you can query debconf to tell you what it is set to)
4. Please enable debug on debconf for the upgrade of grub to see what grub is doing with debconf.

[Po-Hsu Lin (cypressyew)]

Hello,
I'm using MaaS 2.3 (2.3.0-6434-gd354690-0ubuntu1~16.04.1)

For the requested information:
1. curtin config: https://pastebin.canonical.com/p/Ksp7YjYYxB/
2. Installation log for that node: https://pastebin.canonical.com/p/8WKwXxM34V/
3. $ sudo debconf-show debconf
  debconf/priority: high
  debconf-apt-progress/title:
  debconf-apt-progress/info:
  debconf/frontend: Dialog
  debconf-apt-progress/media-change:
  debconf-apt-progress/preparing:
   $ sudo debconf-get-selections
   https://pastebin.canonical.com/p/Sw8jsfcXHw/
4. # DEBIAN_FRONTEND=readline DEBCONF_DEBUG=developer apt-get install grub-efi-amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  grub-pc-bin
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  grub-common grub-efi-amd64-bin grub-efi-amd64-signed grub-pc-bin grub2-common
Suggested packages:
  multiboot-doc grub-emu xorriso desktop-base
The following packages will be upgraded:
  grub-common grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed grub-pc-bin grub2-common
6 upgraded, 0 newly installed, 0 to remove and 27 not upgraded.
Need to get 4,129 kB of archives.
After this operation, 4,096 B of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 grub-efi-amd64-signed amd64 1.66.17+2.02~beta2-36ubuntu3.17 [297 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 grub-efi-amd64 amd64 2.02~beta2-36ubuntu3.17 [65.9 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 grub2-common amd64 2.02~beta2-36ubuntu3.17 [511 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 grub-pc-bin amd64 2.02~beta2-36ubuntu3.17 [888 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 grub-efi-amd64-bin amd64 2.02~beta2-36ubuntu3.17 [661 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 grub-common amd64 2.02~beta2-36ubuntu3.17 [1,705 kB]
Fetched 4,129 kB in 0s (42.3 MB/s)
Preconfiguring packages ...
debconf (developer): starting /tmp/grub-efi-amd64.config.qFN0z0 configure 2.02~beta2-36ubuntu3.16
debconf (developer): <-- SET grub2/linux_cmdline
debconf (developer): --> 0 value set
debconf (developer): <-- SET grub2/linux_cmdline_default
debconf (developer): --> 0 value set
debconf (developer): <-- INPUT medium grub2/linux_cmdline
debconf (developer): --> 30 question skipped
debconf (developer): <-- INPUT medium grub2/linux_cmdline_default
debconf (developer): --> 30 question skipped
debconf (developer): <-- INPUT low grub2/force_efi_extra_removable
debconf (developer): --> 30 question skipped
debconf (developer): <-- INPUT low grub2/update_nvram
debconf (developer): --> 30 question skipped
debconf (developer): <-- GO
debconf (developer): --> 0 ok
(Reading database ... 61346 files and directories currently installed.)
Preparing to unpack .../grub-efi-amd64-signed_1.66.17+2.02~beta2-36ubuntu3.17_amd64.deb ...
Unpacking grub-efi-amd64-signed (1.66.17+2.02~beta2-36ubuntu3.17) over (1.66.16+2.02~beta2...

[Po-Hsu Lin (cypressyew)]

Hi, do we have an update on this?
the new SRU cycle has begun and it is blocking us from testing this node.

Or if I should raise this issue to AMD (if it's BIOS/HW related)

Thanks

[Mathieu Trudel-Lapierre (cyphermox)]

This is possibly a firmware issue, seeing as the default state for grub as provided by maas/curtin config should be to skip modifying nvram.

However, how is the system booting in general? Was BootOrder really set to booting to the network first? If it's set to other devices, and the other device happens to be the hard disk, then shim fallback may cause a new variable to be written to nvram to boot to 'ubuntu', and that variable will always be written as a new thing to boot to first.

[Brian Murray (brian-murray)]

Is there any update on this bug report?

[dann frazier (dannf)]

The system was set to boot from PXE by default, at least before MAAS deployed it:
Feb 22 03:28:41 lodygin cloud-init[3447]: + echo before grub-install efiboot settings
Feb 22 03:28:41 lodygin cloud-init[3447]: before grub-install efiboot settings
Feb 22 03:28:41 lodygin cloud-init[3447]: + efibootmgr
Feb 22 03:28:41 lodygin cloud-init[3447]: BootCurrent: 0002
Feb 22 03:28:41 lodygin cloud-init[3447]: Timeout: 1 seconds
Feb 22 03:28:41 lodygin cloud-init[3447]: BootOrder: 0002,0003,0004
Feb 22 03:28:41 lodygin cloud-init[3447]: Boot0002* UEFI: IP4 Intel(R) Gigabit CT Desktop Adapter
Feb 22 03:28:41 lodygin cloud-init[3447]: Boot0003* UEFI: Built-in EFI Shell
Feb 22 03:28:41 lodygin cloud-init[3447]: Boot0004* Hard Drive

Now, curtin will install an entry and monkey with the boot order during deployment. I'm aware of a couple of firmware implementations where that breaks things - but I don't see signs of those known issues here. And, after deploy you say efibootmgr output looks correct - so that suggests nothing has gone wrong yet.

As @cyphermox mentioned, maas/curtin should be configuring the system to skip modifying NVRAM. And the debconf-get-selections output confirms this:

# Update NVRAM variables to automatically boot into Debian?
grub-efi-amd64 grub2/update_nvram boolean false
grub-pc grub2/update_nvram boolean false
grub2 grub2/update_nvram boolean false

This means efibootmgr output *should be* the same before and after you upgrade GRUB. Can you confirm that is true? Since you mention rebooting as step #4 in reproduction - that suggests the boot order breakage is occuring somewhere in firmware after reboot.

Regarding @cyphermox's point about shim fallback.... these days, curtin leaves the system with an Ubuntu entry, but makes it the 2nd option in the bootorder. The idea is that if MAAS is down for some reason, the system will fallback to booting from the disk (instead of failing to boot altogether). What would happen if PXE failed, and Ubuntu - as the 2nd priority boot option - started? Would shim fallback reorder the menu?

[Po-Hsu Lin (cypressyew)]

Hello,
sorry that I missed this one,

And yes as Dann mentioned here it was set to boot with PXE first.

I can't check the efibootmgr before / after the grub update as the hardware has down, I will attach this bug to our trello card to keep it tracked.

Thanks

[Po-Hsu Lin (cypressyew)]

Hardware long gone, invalidate this bug.