[regression] Cosmic daily images 20180606-11 install but boot only to grub prompt on EFI systems

[Daniel van Vugt]

Cosmic daily images install successfully but then never boot (stuck in grub). I've tried a couple of machines.

Fails: 20180611
Fails: 20180608
Fails: 20180607
Fails: 20180606
Works: 20180530

[Daniel van Vugt (vanvugt)]

Sorry I don't know what other packages to target.

[Daniel van Vugt (vanvugt)]

Here's a diff between my current best known working image and the broken image.

[Daniel van Vugt (vanvugt)]

The only thing that stands out between the working and broken images to me is:

< ii ubuntu-desktop 1.420 amd64 The Ubuntu desktop system
---
> ii ubuntu-desktop 1.421 amd64 The Ubuntu desktop system
1683c1684
< ii ubuntu-minimal 1.420 amd64 Minimal core of Ubuntu
---
> ii ubuntu-minimal 1.421 amd64 Minimal core of Ubuntu
1692c1693
< ii ubuntu-standard 1.420 amd64 The Ubuntu standard system
---
> ii ubuntu-standard 1.421 amd64 The Ubuntu standard system

[Daniel van Vugt (vanvugt)]

Is this legit? Just a mistake?

ubuntu-meta (1.421) cosmic; urgency=medium

  * Refreshed dependencies
  * Moved open-iscsi to server-recommends (LP: #1630946)

 -- root <[e-mail]> Wed, 30 May 2018 07:01:29 +0000

[Mathieu Trudel-Lapierre (cyphermox)]

What daily image are we talking about?

How does "stuck in grub" present? Do you get thrown into a grub prompt, or do you just see nothing after the grub menu disappears?

[Daniel van Vugt (vanvugt)]

We are talking about "cosmic daily images" http://cdimages.ubuntu.com/daily-live/

It's just the grub prompt.

[Jean-Baptiste Lallement (jibel)]

I cannot reproduce this issue in a VM. Could you provide more details about the machines you tried to install on, is it UEFI hardware or with a specific setup?

[Daniel van Vugt (vanvugt)]

It's seemingly got nothing to do with the hardware.

Today I have tested cosmic image 20180611 on two machines:

  * Lenovo X1 Carbon (gen 5)
  * Dell XPS 13 (9343)

and the bug persists. In both cases Ubuntu installs fine, but then the machine never boots.

I'm lucky in that I still have image 20180530 as a workaround.

Any chance I can get access to images 20180531-20180605 to better bisect the bug?

[Daniel van Vugt (vanvugt)]

This is probably coincidence but the working images are slightly bigger than the non-working ones:

-rw------- 1 dan dan 1964244992 May 24 10:46 cosmic-desktop-amd64-20180522.iso
-rw------- 1 dan dan 1964244992 May 26 14:53 cosmic-desktop-amd64-20180525.iso
-rw------- 1 dan dan 1964244992 May 30 15:48 cosmic-desktop-amd64-20180530.iso
-rw------- 1 dan dan 1957183488 Jun 8 12:56 cosmic-desktop-amd64-20180606.iso
-rw------- 1 dan dan 1957183488 Jun 7 15:55 cosmic-desktop-amd64-20180607.iso
-rw------- 1 dan dan 1957183488 Jun 8 15:51 cosmic-desktop-amd64-20180608.iso
-rw------- 1 dan dan 1957183488 Jun 13 12:40 cosmic-desktop-amd64-20180611.iso

[Łukasz Zemczak (sil2100)]

As already asked by Jean-Baptiste, could you give us more info on whether the installation has been done in UEFI or legacy BIOS mode? Is it broken for both cases? We did have a new grub2 released quite recently which might have potentially some effect here - but I am also unable to reproduce the issue when installing on a VM (tried ubuntu-server though). Let me check ubuntu-desktop maybe.

[Daniel van Vugt (vanvugt)]

I am always using UEFI.

I don't think grub2 is the problem. This bug pre-dates that update by about 1 day. And image 20180606, which contains the bug, does not contain the new grub2:

ii grub-common 2.02-2ubuntu8 amd64 GRand Unified Bootloader (common files)
ii grub-gfxpayload-lists 0.7 amd64 GRUB gfxpayload blacklist
ii grub-pc 2.02-2ubuntu8 amd64 GRand Unified Bootloader, version 2 (PC/BIOS version)
ii grub-pc-bin 2.02-2ubuntu8 amd64 GRand Unified Bootloader, version 2 (PC/BIOS binaries)
ii grub2-common 2.02-2ubuntu8 amd64 GRand Unified Bootloader (common files for version 2)

See comment #2.

[Jean-Baptiste Lallement (jibel)]

It's reproducible in a VM with a Cosmic desktop image and UEFI enabled.

[Jean-Baptiste Lallement (jibel)]

Installer log on a UEFI VM

[Łukasz Zemczak (sil2100)]

It seems the subiquity images are not affected. I have first tried the latest cosmic ubuntu-server-live image and was able to boot correctly into the system after installation. Then I have tried the old debian-installer based ubuntu-server images and was able to reproduce the issue, with the system displaying the grub prompt. This explains why I was not able to reproduce the issue before (I was using subiquity). Both on UEFI.

[Łukasz Zemczak (sil2100)]

For now I'm just writing down some of my random thoughts, but looking at Daniel's diff between the working image and non-working, I don't see anything straightforward that could have caused it. That being said, Dimitri had a good idea that one of the livecd-rootfs changes could have caused it. I now see this which might be somehow related?

https://launchpad.net/ubuntu/+source/livecd-rootfs/2.528

Time-wise it could fit, as the package - even though uploaded on the 28th of May, it actually only migrated from cosmic-proposed to cosmic on the 5th of June (our images are built without -proposed enabled). Need to look into the change itself, but it does deal with grub somehow.

[Daniel van Vugt (vanvugt)]

Yes, 'livecd-rootfs' is also a candidate since it doesn't appear in 'dpkg -l' so is hidden from my diff.

[Daniel van Vugt (vanvugt)]

Comment #9 also suggests the problem might have arisen from a change to 'livecd-rootfs'

[Łukasz Zemczak (sil2100)]

I am starting to doubt the theory that livecd-rootfs was responsible. I have built the livefs locally using livecd-rootfs 2.537 (so pre-dating the broken images), repacked the latest cosmic iso to include my squashfs in it, installed ubuntu-server on UEFI and am still experiencing the same grub prompt bug. Either my test-case is invalid or the issue lies somewhere else?

[Łukasz Zemczak (sil2100)]

Looking at the change in livecd-rootfs it seems to be indeed not related, as the modified code-path is only executed for ubuntu-cpc projects.

Writing down some of our thoughts. Me and Jean-Baptiste sat down on the issue just now and noticed a strange thing happening in the ESP. We had 3 installs: A) cosmic ubuntu-server (debian-installer), B) cosmic ubuntu-server-live (subiquity) system and C) bionic ubuntu. A) is obviously broken, we get the grub2 prompt after booting. All cases but A) work, but even for A) it's possible still to boot the system by directly loading the config from the main partition.

We compared the ESP contents in A), B) and C). For B) the grub.cfg file (along with the .efi binaries) is located in the EFI/ubuntu/ directory on the efi partition. For C) there are 2 grub.cfg files present (and the corresponding binaries), one in EFI/ubuntu/ and one in EFI/grub/. Both of those systems boot. However, for A) the ESP only has the EFI/grub/ part. It looks like something that usually installed the required bits to ubuntu/ has changed and now, possibly, this is causing grub not proceeding with its boot process. Problem is we can't seem to see any changes in any of the installer packages that could have touched these parts.

[Łukasz Zemczak (sil2100)]

It looks like the grub2-signed upload is the culprit, looks like the dependency switch we made caused in the d-i cases for grub2-efi-amd64 not to be installed, even though it's required. Will be consulting a fix.

[Jean-Baptiste Lallement (jibel)]

I think it's because grub-efi-amd64 is missing. Installing it in /target at the end of installation (after remounting the efi partition) fixes the problem.

[Łukasz Zemczak (sil2100)]

Since the grub2-signed changes that have introduced this regression are in fact needed, we will be looking into fixing this in the installer.

[Łukasz Zemczak (sil2100)]

I would need someone to take a look and review my proposed fix for debian-installer. I have tested that on a custom ISO image and as a result I got my system booting. It did cause the grub prompt to appear (asking on which partition to install), but I suppose that's not anything new.

[Jean-Baptiste Lallement (jibel)]

@sil2100, Colin already commented on a similar change you proposed against ubiquity https://code.launchpad.net/~sil2100/ubiquity/+git/ubiquity/+merge/348201/comments/909076

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub-installer - 1.128ubuntu9

---------------
grub-installer (1.128ubuntu9) cosmic; urgency=medium

  * grub-installer: install grub-pc for EFI setups and make sure we're
    not purging it earlier in case it got installed. It's needed to make sure
    the right maintainer scripts are run and the ESP populated. (LP: #1775743)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 19 Jun 2018 10:25:14 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubiquity - 18.10.4

---------------
ubiquity (18.10.4) cosmic; urgency=medium

  [ Chen-Han Hsiao (Stanley) ]
  * Add efivars to mountpoints loaded at bootloader install time.
    (LP: #1772374)

  [ Steve Langasek ]
  * scripts/plugininstall.py: don't hard-code a resume partition in
    /etc/initramfs-tools/conf.d/resume at install time. In bionic and later,
    initramfs-tools will autodetect an appropriate resume partition at
    initramfs generation time, so ubiquity's resume setting is redundant and
    possibly wrong. LP: #1768230.

  [ Łukasz 'sil2100' Zemczak ]
  * Make sure that grub-pc is not removed after installation for both EFI and
    legacy BIOS cases as we now technically need it even for EFI installs.
    (LP: #1775743)

  [ Mathieu Trudel-Lapierre ]
  * Automatic update of included source packages: grub-installer
    1.128ubuntu9.

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 25 Jun 2018 13:57:40 -0400

[oldfred (oldfred)]

Not sure proposed fix is all of issue.

I boot 18.04 by default with UEFI and every test or experimental install in the past has overwritten my /EFI/ubuntu.
But install of Cosmic did not overwrite my /EFI/ubuntu but created /EFI/grub. It has a correct /EFI/grub/grub.cfg, but my UEFI is set to default boot using /EFI/ubuntu.
But booting grub UEFI entry booted me into my default install.
Issue is similar to previous bug reports where a change to GRUB_DISTRIBUTOR= creates a new UEFI entry, but internally either efi_distributor or perhaps just the code that defines where to find grub is hard coded to /EFI/ubuntu/grub.cfg.
It needs to use grub.cfg in the ESP folder it creates. And kluge on Kubuntu should finally be fixed.

Older version, not sure what you have now.

./.pc/install_arm64_naming.patch/util/grub-install.c: efi_distributor = "ubuntu";
./debian/patches/install_efi_ubuntu_flavours.patch:+ efi_distributor = "ubuntu";
./ChangeLog: GRUB_DISTRIBUTOR on efibootmgr line.
./debian/default/grub:GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
./debian/postinst.in: bootloader_id="$(config_item GRUB_DISTRIBUTOR | tr A-Z a-z | \
./debian/changelog: - Adjust UEFI installation to cope with Kubuntu setting GRUB_DISTRIBUTOR

[Steve Langasek (vorlon)]

On Tue, Jun 26, 2018 at 09:31:24PM -0000, Fred Palmer wrote:
> But install of Cosmic did not overwrite my /EFI/ubuntu but created
> /EFI/grub. It has a correct /EFI/grub/grub.cfg, but my UEFI is set to
> default boot using /EFI/ubuntu.

With which daily image did you see this?

[Daniel van Vugt (vanvugt)]

AFAICS, no image has been released with both fixes yet:
  http://cdimages.ubuntu.com/daily-live/

[oldfred (oldfred)]

> With which daily image did you see this?

The install of Cosmic has modified date of Mon 11 Jun 2018 07∶55∶53 AM CDT.
Not sure when I downloaded it.

[Daniel van Vugt (vanvugt)]

AFAIK, all June images older than 20180627 will have the bug.

Please wait until the fix for this bug is released before retesting.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1775743

[oldfred (oldfred)]

Just saw one thread in Ubuntu Forums & one in AskUbuntu where with 18.04 the Boot-Repair output from efibootmgr -v shows no ubuntu entry and only has a grub entry.

See:
https://ubuntuforums.org/showthread.php?t=2396042
And:
https://askubuntu.com/questions/1053895/system-always-boots-to-grub-prompt?noredirect=1#comment1722812_1053895
This user already reported that copying /EFI/grub/grub.cfg to /EFI/ubuntu/grub.cfg fixed his boot issue.

[Sean Clarke (belsean21)]

From above comment

https://ubuntuforums.org/showthread.php?t=2396042

Installed from latest Ubuntu Studio 18.04 ISO.

Boots to grub> prompt

Also fixed by copying /boot/efi/EFI/grub/grub.cfg to /boot/efi/EFI/ubuntu/

[Daniel van Vugt (vanvugt)]

This bug has been fixed for a while already and is about cosmic only (Ubuntu 18.10).

If you have any problems with 18.04 then please open a new bug.

[goahead97 (goahead97)]

Hello

I need to install Ubuntu 18.04.4 LTS in the following machine that has UEFI but I only get a blank screen after selecting the "intall" option or the "try Ubuntu live" option.

Processor Manufacturer Intel® Processor Type Core™ i5 Processor Model i5-1035G1 Processor Speed 1 GHz Processor Core Quad-core (4 Core™) Display & Graphics Graphics Controller Manufacturer Intel® Graphics Controller Model UHD Graphics Graphics Memory Technology DDR4 SDRAM Graphics Memory Accessibility Shared Memory Standard Memory 8 GB Memory Technology DDR4 SDRAM

I saw the solution suggested in the post#21: "I think it's because grub-efi-amd64 is missing. Installing it in /target at the end of installation (after remounting the efi partition) fixes the problem."

I wanted the mentioned solution but I do not know how to install grub-efi-amd64 in the USB stick that has the Ubuntu 18.04.4 LTS image that I want to install.

Does any of you know how this grub-efi-amd64 can be installed along the Ubuntu 18.04.4 LTS image?

Thanks