update_nvram ignored when installing Grub on Debian+UEFI

[Jonathan Heathcote]

When the grub/update_nvram config parameter is set to False, Curtin usually passes the --no-nvram argument to grub-install, preventing it from adding any UEFI boot entries. When running on a Debian-based distribution and a UEFI boot, Curtin instead uses the grub-multi-install wrapper script. Unfortunately, the --no-nvram argument is not passed in this case.

The impact of this bug is that the newly installed boot loader will unconditionally be inserted at the top of the boot order. Aside from being unexpected behaviour, this can cause issues on systems with buggy UEFI firmware.

Relevant code: curtin/commands/install_grub.py:298

    if grub_cmd == GRUB_MULTI_INSTALL:
        # grub-multi-install is called with no arguments
        install_cmds.append([grub_cmd])
    elif grub_cmd:
        install_cmds.append(
            [grub_cmd, '--target=%s' % grub_target,
             '--efi-directory=%s' % efidir, '--bootloader-id=%s' % bootid,
             '--recheck'] + ([] if update_nvram else ['--no-nvram']))

Note that the grub-multi-install script forwards all arguments given to its calls to grub-install. Adding a --no-nvram argument to the grub-multi-install command (when required) should fix this bug.

[Malo Paletou (mpaletou)]

@mossblaser did you found a workaround by any chance for this bug ?

We are having troubles while autoinstall because of this (UEFI PXE boot) and we are stuck as we don't know how to workaround this bug

[Jonathan Heathcote (mossblaser)]

Yes -- although it is pretty gross...

Essentially we use a series of 'early-commands' to patch Curtin inside the installer environment. This is icky enough, but since Curtin lives in a snap you also need to work-around snap's pseudo-immutability... See the snippet below:

    autoinstall:

      # ...

      early-commands:
        # Workaround for Curtin bug in which grub.update_nvram is ignored during
        # grub installation causing the UEFI boot order to be changed.
        #
        # https://bugs.launchpad.net/curtin/+bug/2002085
        #
        # In this work-around we modify the grub-multi-install script to pretend
        # that it has been given the --no-nvram argument even though Curtin has
        # forgotten to do this.
        #
        # NB: This change is made in the ephemeral installer environment and not to
        # the installed system.
        #
        # Because Curtin (and its copy of grub) live inside a snap's read-only
        # filesystem, to patch it in-place is a faff. The trick used here is to
        # mount a (writeable) tmpfs filesystem on-top of the directory containing
        # grub-multi-install.
        - cp -r `dirname \`which curtin\``/../usr/lib/grub/ /tmp/usr_lib_grub/
        - mount -t tmpfs tmpfs `dirname \`which curtin\``/../usr/lib/grub/
        - sed -i.old '1 aset -- --no-nvram "$@"' /tmp/usr_lib_grub/grub-multi-install
        - cp -r /tmp/usr_lib_grub/* `dirname \`which curtin\``/../usr/lib/grub/

In case you're not aware of the semi-related bug #1714090 (GRUB will re-write the UEFI boot order when its package is upgraded), you may also wish to add the following which tells the GRUB package not to do this:

      debconf_selections:
        # Prevent updates to the grub2 package triggering a reordering of the
        # UEFI boot order.
        #
        # This default behaviour (rewriting the UEFI boot order whenever grub is
        # updated) is a bug in Debian/Ubuntu's packaging of grub being tracked
        # here: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1714090
        disable_grub_nvram_edits_on_update: |
          grub2 grub2/update_nvram boolean false

The latter, of course, is not necessary to make your autoinstall do what you want but will hopefully save you from a nasty surprise later!

[Malo Paletou (mpaletou)]

Many thanks for sharing Jonathan.

Maybe we have a different curtin version, as on my side changes are effective with your patch in the snap environment but I am still getting the same error.

By checking the failing command, it looks like it is executed directly from target install... Which is not patched:

unshare --fork --pid -- chroot /target /usr/lib/grub/grub-multi-install

I may be wrong but I didn't found any way to patch it into target as early and post commands can't help us with that.