EFI booting + /boot on LVM == inaccessible boot menu

[Steve Langasek]

[Impact]
This issue makes it impossible for UEFI users to access to boot menu and choose their kernel if /boot is installed on LVM.

[Test case]
1) Install Ubuntu on UEFI; put /boot on a LVM LV.
2) Reboot after the install
3) Verify that you will get a GRUB menu at boot.

[Regression potential]
Give particular attention to boot behavior on different disk setups: failure to show the menu on LVM or other disk setups when holding the SHIFT key, or showing the menu at every boot unnecessarily on other types of disk configurations: /boot on a physical partition, etc.

---

Because EFI does not support instantaneous read of modifier keys (i.e. holding down shift at boot), the only way to reliably show a boot menu is by letting the system boot, interrupting the boot, and letting the menu be displayed because 'recordfail' has been set.

If /boot/grub is on LVM, recordfail does not work, because grub doesn't have write support on LVM, so saveenv doesn't work. Indeed, from the grub.cfg on such a system, the recordfail function is written as:

function recordfail {
  set recordfail=1
  # GRUB lacks write support for lvm, so recordfail support is disabled.
}

The interaction of these two limitations means that systems with /boot/grub on LVM cannot reliably get a grub menu, ever.

While I think that in the long term we should always put /boot/grub on the ESP (which means it will always be writable by grub), which is in fact what we did for Ubuntu Core, in the meantime I believe what we need to do here is always show the boot menu if we are booted under EFI and we have a non-writable grubenv.

[Julian Andres Klode (juliank)]

JFTR so I don't forget to mention it: I don't think putting /boot/grub on the ESP is a good idea, as it makes things so much harder if you want to install multiple distributions on one system; unless you put it in a vendor, or better, machine-id, specific subdirectory.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02+dfsg1-5ubuntu9

---------------
grub2 (2.02+dfsg1-5ubuntu9) disco; urgency=medium

  [ Steve Langasek ]
  * debian/patches/quick-boot-lvm.patch: If we don't have writable
    grubenv and we're on EFI, always show the menu. Closes LP: #1800722.

  [ Mathieu Trudel-Lapierre ]
  * debian/patches/mkconfig_leave_breadcrumbs.patch: make sure grub-mkconfig
    leaves a trace of what files were sourced to help generate the config
    we're building.

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 07 Jan 2019 17:32:01 -0500

[Brian Murray (brian-murray)]

Hello Steve, or anyone else affected,

Accepted grub2 into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02+dfsg1-5ubuntu8.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Steve, or anyone else affected,

Accepted grub2-signed into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.110.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Steve, or anyone else affected,

Accepted grub2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02-2ubuntu8.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Steve, or anyone else affected,

Accepted grub2-signed into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.93.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mathieu Trudel-Lapierre (cyphermox)]

Verification-done for bionic with grub2 / grub2-signed:

ubuntu@ubuntu:~$ dpkg -l grub-efi\* | grep ii | awk '{print $2" "$3 }'
grub-efi-amd64 2.02-2ubuntu8.10
grub-efi-amd64-bin 2.02-2ubuntu8.10
grub-efi-amd64-signed 1.93.11+2.02-2ubuntu8.10

GRUB correctly shows the menu at every boot when the system is installed with full-disk LVM on EFI: /boot is on / which is on LVM, only /boot/efi is on a primary partition.

[Mathieu Trudel-Lapierre (cyphermox)]

Verification-done on cosmic for grub2 / grub2-signed:

ubuntu@ubuntu:~$ dpkg -l grub-efi\* | grep ii | awk '{ print $2" "$3 }'
grub-efi-amd64 2.02+dfsg1-5ubuntu8.1
grub-efi-amd64-bin 2.02+dfsg1-5ubuntu8.1
grub-efi-amd64-signed 1.110.1+2.02+dfsg1-5ubuntu8.1

Menu shows as expected, since the system is installed with full-disk LVM on EFI.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for grub2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02+dfsg1-5ubuntu8.1

---------------
grub2 (2.02+dfsg1-5ubuntu8.1) cosmic; urgency=medium

  [ Mathieu Trudel-Lapierre ]
  * debian/grub-check-signatures: check kernel signatures against keys known
    in firmware, in case a kernel is signed but not using a key that will pass
    validation, such as when using kernels coming from a PPA. (LP: #1789918)
  * debian/patches/mkconfig_leave_breadcrumbs.patch: make sure grub-mkconfig
    leaves a trace of what files were sourced to help generate the config
    we're building. (LP: #1812863)

  [ Steve Langasek ]
  * debian/patches/quick-boot-lvm.patch: If we don't have writable
    grubenv and we're on EFI, always show the menu. Closes LP: #1800722.

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 22 Jan 2019 09:57:07 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02-2ubuntu8.10

---------------
grub2 (2.02-2ubuntu8.10) bionic; urgency=medium

  [ Mathieu Trudel-Lapierre ]
  * debian/grub-check-signatures: check kernel signatures against keys known
    in firmware, in case a kernel is signed but not using a key that will pass
    validation, such as when using kernels coming from a PPA. (LP: #1789918)
  * debian/patches/mkconfig_leave_breadcrumbs.patch: make sure grub-mkconfig
    leaves a trace of what files were sourced to help generate the config
    we're building. (LP: #1812863)

  [ Steve Langasek ]
  * debian/patches/quick-boot-lvm.patch: If we don't have writable
    grubenv and we're on EFI, always show the menu. Closes LP: #1800722.

 -- Mathieu Trudel-Lapierre <email address hidden> Wed, 09 Jan 2019 14:04:09 -0500

[m4t (m4t)]

Seeing a regression here on 18.10/amd64 with 2.02+dfsg1-5ubuntu8.1 using EFI+btrfs boot volume. No LVM used, just GPT on NVMe.

Regardless of my settings in /etc/default/grub, the boot wait is now 30 seconds.

Relevant settings:
GRUB_HIDDEN_TIMEOUT=1
GRUB_HIDDEN_TIMEOUT_QUIET=false
GRUB_TIMEOUT=0

I've downgraded using the following commands:
apt install grub-common=2.02+dfsg1-5ubuntu8 grub-efi-amd64=2.02+dfsg1-5ubuntu8 grub-efi-amd64-bin=2.02+dfsg1-5ubuntu8 grub2-common=2.02+dfsg1-5ubuntu8
apt-mark hold grub2-common grub-efi-amd64

And it's returned the previous behavior.

[Klaus Kudielka (kkudielka)]

I see a similar behaviour on my EFI system with BTRFS root on bionic, which recently upgraded to:

grub-common/bionic-updates,now 2.02-2ubuntu8.10 amd64

GRUB_TIMEOUT is overridden to be 30 seconds.
GRUB_TIMEOUT_STYLE=hidden is also overridden.

Workaround for the timeout in /etc/default/grub: GRUB_RECORDFAIL_TIMEOUT=$GRUB_TIMEOUT

[Diftraku (diftraku)]

I experienced the regression mentioned in earlier comments on my freshly installed and updated Ubuntu Server 18.04 with 2.02-2ubuntu8.12 with EFI and boot on LVM.

Dowgrading to 2.02-2ubuntu8 restores previously expected behaviour in regards to hidden timeout and timeout.

An older 18.04 install running 2.02-2ubuntu8.9 still has the old, expected behaviour.

[Steve Langasek (vorlon)]

@Diftraku you will need to be specific about what behavior you are seeing and why you think this is a bug, since the behavior of 2.02-2ubuntu8.12 is known to not be the same as in 2.02-2ubuntu8.10.

If you have /boot on LVM and are booting in UEFI mode, then you will see a boot delay.

[Diftraku (diftraku)]

@vorlon When grub-common, grub-efi-amd64, grub-efi-amd64-bin and grub2-common are installed with version 2.02-2ubuntu8.12 the following configuration options (in /etc/default/grub) are not honoured:
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_RECORDFAIL_TIME=0

The resulting "grub.cfg" has following lines at the end of 00_header-block:
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=hidden
    set timeout=0
  # Fallback hidden-timeout code in case the timeout_style feature is
  # unavailable.
  elif sleep --interruptible 0 ; then
    set timeout=0
  fi
fi
if [ $grub_platform = efi ]; then
  set timeout=30
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
  fi
fi

These lines override any timeouts or menu styles set in the configuration (via /etc/default/grub). The result is a 30 second timeout with a menu when booting grub. Having no recordfail support would default to 30 second timeout but running on EFI forces the timeout style to "menu" as well.

When downgrading to version 2.02-2ubuntu8.8 equivalents of the packages mentioned above, "grub.cfg" has the following output at the end of the 00_header block:
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=hidden
    set timeout=0
  # Fallback hidden-timeout code in case the timeout_style feature is
  # unavailable.
  elif sleep --interruptible 0 ; then
    set timeout=0
  fi
fi

Absent is the "platform == efi" block that forces timeout to 30 seconds and timeout style to menu. This results in a quiet startup for grub without any boot delay.

The main issue is 2.02-2ubuntu8.12 (I have not been able to install 2.02-2ubuntu8.10 from the repos directly) overrides GRUB_TIMEOUT_STYLE and GRUB_TIMEOUT without any warning. This means that GRUB_HIDDEN_TIMEOUT and GRUB_HIDDEN_TIMEOUT_QUIET are also never used. The previous behaviour is as described above, where the quiet boot happens without boot delay.

I indeed have /boot on LVM on an EFI system but what I want from grub is quiet boot with no timeout, which the 2.02-2ubuntu8.8 version provides. I feel this is a regression due to how earlier behaviour is no longer happening as expected nor are there any warnings about this when running update-grub. Running on EFI and LVM is not an excuse to silently ignore configured defaults. There should at least be a warning that the configured defaults are going to be ignored.

I have not tried to see if you only need to downgrabe grub-common or grub2-common for the old behaviour to return.

[Diftraku (diftraku)]

Pardon for the double comment: I meant 2.02-2ubuntu8 when mentioning 2.02-2ubuntu8.8.

[Steve Langasek (vorlon)]

On Fri, Mar 01, 2019 at 12:42:47PM -0000, Diftraku wrote:
> @vorlon When grub-common, grub-efi-amd64, grub-efi-amd64-bin and grub2-common are installed with version 2.02-2ubuntu8.12 the following configuration options (in /etc/default/grub) are not honoured:
> GRUB_TIMEOUT_STYLE=hidden
> GRUB_TIMEOUT=0
> GRUB_RECORDFAIL_TIME=0

The correct spelling of this last option is GRUB_RECORDFAIL_TIMEOUT, not
GRUB_RECORDFAIL_TIME.

[Diftraku (diftraku)]

Fixing the spelling on the recordfail timeout changes the timeout on both the recordfail and efi blocks:
if [ "${recordfail}" = 1 ] ; then
  set timeout=0
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=hidden
    set timeout=0
  # Fallback hidden-timeout code in case the timeout_style feature is
  # unavailable.
  elif sleep --interruptible 0 ; then
    set timeout=0
  fi
fi
if [ $grub_platform = efi ]; then
  set timeout=0
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
  fi
fi

This still leaves `GRUB_TIMEOUT_STYLE=hidden`, as the timeout style is forcefully set to `menu` if running on EFI. The menu indeed does not show when both the normal and recordfail timeouts are set to 0 (as expected).

[Tom Chiverton (bugs-launchpad-net-falkensweb)]

Is there a seperate bug for having to add GRUB_RECORDFAIL_TIMEOUT=$GRUB_TIMEOUT on EFI systems ?