Latest update causes 30 sec. menu delay timeout

[oshunluvr]

[SRU Justification]
There is a behavior regression on non-EFI systems with the latest SRU of grub2 which causes unnecessary boot delays.

[Test case]
1. Install a system using BIOS mode with /boot on LVM.
2. Reboot and verify that the boot menu is shown for 30 seconds at boot.
3. Install grub* from -proposed.
4. Reboot and verify that the boot menu is not shown.
5. Install a system using UEFI mode with /boot on LVM.
6. Reboot and verify that the boot menu is shown for 30 second at boot.
7. Install grub* from -proposed.
8. Reboot and verify that the boot menu is still shown.

[Regression potential]
The test case is sufficient to verify all possible paths work correctly after the SRU.

KDEneon LTS user edition 5.12, Release 18.04

grub-common 2.02-2ubuntu8.10 from Ubuntu updates

Since latest update near the end of Jan. 2019 (29th?), grub menu has 30 second delay every reboot as though recordfail is set. It did not do this before the update.

After messing with grubenv and recordfails settings, I found this paragraph in 00_header in 18.04 that is not in 16.04:

if lsefi; then
  set timeout=30
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
  fi
fi

Note "timeout=30" which is the same as the recordfail timeout. As a test, I edited /boot/grub/grub.cfg and changed the 30 to 17 and sure enough, that's the source of the timeout.

Here's the paragraph from 00_header:

if lsefi; then set timeout=${GRUB_RECORDFAIL_TIMEOUT:-30}
  if [ x\$feature_timeout_style = xy ] ; then
    set timeout_style=menu
  fi
fi

One odd thing is the 00_header file is from Jan 9th but this bug has just appeared so it must be related to grub-common not the the header file.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[m4t (m4t)]

I believe it had to do with the patch debian/patches/quick-boot-lvm.patch as described here: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1800722

Here's what I wrote in a comment (#12) there a couple days ago:

Seeing a regression here on 18.10/amd64 with 2.02+dfsg1-5ubuntu8.1 using EFI+btrfs boot volume. No LVM used, just GPT on NVMe.

Regardless of my settings in /etc/default/grub, the boot wait is now 30 seconds.

Relevant settings:
GRUB_HIDDEN_TIMEOUT=1
GRUB_HIDDEN_TIMEOUT_QUIET=false
GRUB_TIMEOUT=0

I've downgraded using the following commands:
apt install grub-common=2.02+dfsg1-5ubuntu8 grub-efi-amd64=2.02+dfsg1-5ubuntu8 grub-efi-amd64-bin=2.02+dfsg1-5ubuntu8 grub2-common=2.02+dfsg1-5ubuntu8
apt-mark hold grub2-common grub-efi-amd64

And it's returned the previous behavior.

[Steve Langasek (vorlon)]

It is expected behavior that, after this SRU, some systems which are configured in such a way that /boot/grub is not writable by grub will take longer to boot. This is a boot speed regression, but it is not a functional regression; no systems should be failing to boot as a result of this change.

The change is deliberate and necessary because without it, it is impossible on a UEFI system to reliably access the boot menu. That means that if you ever had a kernel that fails to boot, you will be stuck with a non-bootable system and be unable to recover it from the grub menu (by booting the previous, working kernel).

So I believe this bug is 'wontfix'.

[Steve Langasek (vorlon)]

As a user you do have the option of adjusting the timeout by setting GRUB_RECORDFAIL_TIMEOUT to a different value in /etc/default/grub. I strongly recommend that you not set it to 0.

[Steve Langasek (vorlon)]

Also,

> using EFI+btrfs boot volume. No LVM used, just GPT on NVMe.

Yes, this change was introduced to address the problem as seen on LVM-backed /boot/grub, but that is certainly not the only configuration for which GRUB has read support but lacks write support. The fix is generic to all such filesystems that are read-only in GRUB, and that is by design.

[oshunluvr (stuartksmith)]

Not using EFI here at all; BTRFS+GPT but no EFI. IMO forcing reference to EFI when it doesn't exist is poor programming for a universal tool like a boot manager. EFI should be included but gracefully enough not to cause problems. If the strong recommendation is to not set GRUB_RECORDFAIL_TIMEOUT to zero, why is it acceptable to delay boot for functionality that doesn't exist?

Seems at a minimum the 00_header should be re-written to bypass the "lsefi" stanza if EFI is not in use.

[Steve Langasek (vorlon)]

On Mon, Feb 04, 2019 at 04:47:35PM -0000, oshunluvr wrote:
> Seems at a minimum the 00_header should be re-written to bypass the
> "lsefi" stanza if EFI is not in use.

The whole point of the 'lsefi' command is to check at runtime whether EFI is
in use. If you have not booted the EFI GRUB, the lsefi command should not
exist and this check should return false.

If that is not the case, then we have more investigation to do.

[Steve Langasek (vorlon)]

grub> if lsefi; then
> set foo=bar
> fi
error: can't find command `lsefi'.
grub> echo $foo
bar

Indeed, this is not working as expected. I'll work on finding another option that lets us properly detect EFI at runtime.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02+dfsg1-5ubuntu11

---------------
grub2 (2.02+dfsg1-5ubuntu11) disco; urgency=medium

  [ Mathieu Trudel-Lapierre ]
  * debian/grub-check-signatures: properly account for DB showing as empty on
    some broken firmwares: Guard against mokutil --export --db failing, and do
    a better job at finding the DER certs for conversion to PEM format.
    (LP: #1814575)

  [ Steve Langasek ]
  * debian/patches/quick-boot-lvm.patch: checking the return value of
    'lsefi' when the command doesn't exist does not do what's expected, so
    instead check the value of $grub_platform which is simpler anyway.
    LP: #1814403.

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 04 Feb 2019 17:51:15 -0500

[Brian Murray (brian-murray)]

Hello oshunluvr, or anyone else affected,

Accepted grub2 into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02+dfsg1-5ubuntu8.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello oshunluvr, or anyone else affected,

Accepted grub2-signed into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.110.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

Hello oshunluvr, or anyone else affected,

Accepted grub2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02-2ubuntu8.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello oshunluvr, or anyone else affected,

Accepted grub2-signed into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.93.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Felipe Castillo (fcastillo.ec)]

I just tested the packages in proposed for Ubuntu Bionic and I still see 30 seconds wait times after boot.
I have a particular configuration, which was done automatically by Ubuntu when I freshly installed the system a while back.
I have an NVMe drive with two partitions, one is an LVM partition for the root (/), which includes /boot (except for /boot/efi). The second partition has /boot/efi. I remember Ubuntu did this because grub or something else might have trouble reading (or writing?) to an LVM partition and it needed this additional partition and it had to be at the beginning (which it is).

So, I have an NVMe with LVM, which is still making me see these errors.

[Steve Langasek (vorlon)]

Hello oshunluvr, or anyone else affected,

Accepted grub2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02-2ubuntu8.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello oshunluvr, or anyone else affected,

Accepted grub2-signed into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.93.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Jani Uusitalo (uusijani)]

Fixed for me by the latest -proposed package. I'm running Bionic, non-EFI, and was affected by the issue with the earlier package.

[Dennis Nehen (dnehen)]

Unwanted side effect: I have had my GRUB_TIMEOUT set to 300 so that grub would wait for 5 minutes. Now that setting seems to have no effect. The delay is now just 30 sec.

[Mathieu Trudel-Lapierre (cyphermox)]

@Dennis; when running the upgrade your GRUB_TIMEOUT may have been reset, depending on what options you picked when prompted about the config change.

Verification-done for bionic using grub2/2.02-2ubuntu8.12, grub2-signed/1.93.13:

I have run installs both with and without LVM on UEFI and on legacy BIOS; as expected UEFI setups with LVM get a menu at every boot, and non-LVM setup do not get one and instead expect the user to hit ESC at early boot to get a grub menu.

[Michael Shannon (mrshannon-aerospace)]

Verification-done for bionic using grub2/2.02-2ubuntu8.12 and grub2-signed/1.93.13+2.02-2ubuntu8.12:

This fixes this issue for UEFI with /boot/grub on ZFS.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02-2ubuntu8.12

---------------
grub2 (2.02-2ubuntu8.12) bionic; urgency=medium

  * debian/grub-check-signatures: make sure grub-check-signatures conserves
    its execute bit.

grub2 (2.02-2ubuntu8.11) bionic; urgency=medium

  [ Mathieu Trudel-Lapierre ]
  * debian/grub-check-signatures: properly account for DB showing as empty on
    some broken firmwares: Guard against mokutil --export --db failing, and do
    a better job at finding the DER certs for conversion to PEM format.
    (LP: #1814575)
  * debian/patches/linuxefi_disable_sb_fallback.patch: Disallow unsigned
    kernels if UEFI Secure Boot is enabled. If UEFI Secure Boot is enabled
    and kernel signature verification fails, do not boot the kernel. Patch
    from Linn Crosetto. (LP: #1401532)

  [ Steve Langasek ]
  * debian/patches/quick-boot-lvm.patch: checking the return value of
    'lsefi' when the command doesn't exist does not do what's expected, so
    instead check the value of $grub_platform which is simpler anyway.
    LP: #1814403.

 -- Mathieu Trudel-Lapierre <email address hidden> Thu, 07 Feb 2019 18:20:04 -0500

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for grub2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.93.13

---------------
grub2-signed (1.93.13) bionic; urgency=medium

  * Rebuild against grub2 2.02-2ubuntu8.12.

grub2-signed (1.93.12) bionic; urgency=medium

  * Rebuild against grub2 2.02-2ubuntu8.11.
    (LP: #1401532) (LP: #1814403) (LP: #1814575)

 -- Mathieu Trudel-Lapierre <email address hidden> Thu, 07 Feb 2019 19:28:09 -0500

[Mathieu Trudel-Lapierre (cyphermox)]

Verification-done on cosmic with grub2/2.02+dfsg1-5ubuntu8.2, grub2-signed/1.110.2:

As expected, LVM in UEFI leads to a GRUB menu at boot, since it would otherwise not be available. For LVMs on legacy BIOS, this does not happen. I find this SRU is working correctly.

[oshunluvr (stuartksmith)]

Thanks for the quick fix.

[Ian McMichael (ian-sigma-uk)]

I have an 18.04.2 desktop installation running grub2 2.02-2ubuntu8.12, which is still pausing for 30 seconds at the grub menu on every boot.

The system is UEFI and running secure boot, with nVidia binary drivers if that matters? The disk is an NVMe drive with two partitions. The first is mounted as /boot/efi and the rest an LVM PV containing root and swap LVs.

I could edit /etc/grub.d/00_header to change the following code:

if [ $grub_platform = efi ]; then
  set timeout=30
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
  fi
fi

However this system used to boot normally without the 30 second delay at the grub menu and any changes I make will be overwritten by a future grub2 package update.

Is there anything else I can do to get the system to boot without the grub menu appearing? Can I provide any more information about my system to help debug this further?

[RussianNeuroMancer (russianneuromancer)]

I installed 2.02+dfsg1-5ubuntu8.2 on fresh 18.10 and I still have 30 seconds counter.

SSD layout is GPT with small esp partition, btrfs partition for @, and swap.

[oshunluvr (stuartksmith)]

Two immediately above comments reference EFI boot. This bug report and fix were regarding an original bug that appeared in non-EFI boot environment.

It's possible your encountering a different bug or have an unclean EFI

[Mathieu Trudel-Lapierre (cyphermox)]

You absolutely can change /etc/grub.d/00_header to change or remove the code. The problem is; I don't think there is a way around us providing a way to reach the menu for the those setups where "recordfail", the feature that lets the boot menu start on failure, doesn't work, while also making it not show every boot.

In other words, your setup works right now, but waits thirty seconds to boot -- that timeout can be adjusted by setting GRUB_RECORDFAIL_TIMEOUT=<some value> in /etc/default/grub. Reducing the timeout value is probably the best solution in all cases, but I do think the default of 30 seconds is reasonable. You get enough time to see the menu and respond, etc. And it doesn't hold up the boot very long in the grand scheme of things.

When your system doesn't boot correctly through the kernel however, it would likely not show the menu without this fix -- you'd have no way to switch back to a working kernel, because the menu can be hard to reach at all in these setups.

This bug was meant to address an issue that caused unnecessary timeouts on systems that should not have needed this workaround, due to some unforeseen limitations in GRUB. We believe it's been addressed appropriately.

If you still feel your system is showing the menu and timeout at every boot unnecessarily, or if you feel the default timeout is too long, please file a separate bug with the information specific to your system and particular case so we can look into it -- the particularities of the system will be important to include in such a bug report.

[RussianNeuroMancer (russianneuromancer)]

> If you still feel your system is showing the menu and timeout at every boot unnecessarily, or if you feel the default timeout is too long, please file a separate bug with the information specific to your system and particular case so we can look into it

In this case please look into https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1815002 for efi+btrfs combination.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02+dfsg1-5ubuntu8.2

---------------
grub2 (2.02+dfsg1-5ubuntu8.2) cosmic; urgency=medium

  [ Mathieu Trudel-Lapierre ]
  * debian/grub-check-signatures: properly account for DB showing as empty on
    some broken firmwares: Guard against mokutil --export --db failing, and do
    a better job at finding the DER certs for conversion to PEM format.
    (LP: #1814575)

  [ Steve Langasek ]
  * debian/patches/quick-boot-lvm.patch: checking the return value of
    'lsefi' when the command doesn't exist does not do what's expected, so
    instead check the value of $grub_platform which is simpler anyway.
    LP: #1814403.

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 05 Feb 2019 11:05:56 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.110.2

---------------
grub2-signed (1.110.2) cosmic; urgency=medium

  * Rebuild against grub2 2.02+dfsg1-5ubuntu8.2. (LP: #1814403) (LP: #1814575)

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 05 Feb 2019 11:19:47 -0500

[Luke (lukethenuke)]

I just encountered the issue that the parameter GRUB_TIMEOUT in /etc/default/grub does not reduce the timeout for uefi boot anymore. I was able to pinpoint the issue in the grub config to line 98 [1]. From your issue here I understand that this is the way it is supposed to work.

This should be documented somewhere as the documentation does not mention that this setting does not work for uefi boot [2].

As a workaround you could edit /etc/grub.d/00_header to check in line 387 if the timeout is less than 5 [3].

[1]
/etc/default/grub ~
GRUB_TIMEOUT=5
/boot/grub/grub.cfg:98 ~
if [ $grub_platform = efi ]; then
  set timeout=30 # this value is the issue
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
  fi
fi

[2]
info -f grub -n 'Simple configuration' ~
'GRUB_TIMEOUT'
     Boot the default entry this many seconds after the menu is
     displayed, unless a key is pressed. The default is '5'. Set to
     '0' to boot immediately without displaying the menu, or to '-1' to
     wait indefinitely.

[3]
/etc/grub.d/00_header:387 ~
- set timeout=${GRUB_RECORDFAIL_TIMEOUT:-30}
+ set timeout=${$(if (( $GRUB_TIMEOUT < 5 )); then echo "5"; else echo "$GRUB_TIMEOUT"; fi):-30}

[Andrew G (andrew700)]

I am experiencing this problem but am new to ubuntu and linux.

As above 'This bug was fixed in the package grub2-signed - 1.110.2', please could someone provide some basing instructions for downloading and updating grub2?

Many thanks

[zyf0330 (zyf0330)]

I have this problem too like andrew700. I am on Ubuntu 20.04.
exec "apt show grub2-signed" shows that "No packages found".