[regression] Cosmic daily images 20180606-11 install but boot only to grub prompt on EFI systems

Bug #1775743 reported by Daniel van Vugt on 2018-06-08
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
grub-installer (Ubuntu)
Critical
Łukasz Zemczak
grub2 (Ubuntu)
Critical
Unassigned
grub2-signed (Ubuntu)
Critical
Unassigned
ubiquity (Ubuntu)
Critical
Łukasz Zemczak

Bug Description

Cosmic daily images install successfully but then never boot (stuck in grub). I've tried a couple of machines.

Fails: 20180611
Fails: 20180608
Fails: 20180607
Fails: 20180606
Works: 20180530

Related branches

Sorry I don't know what other packages to target.

Changed in grub-installer (Ubuntu):
importance: Undecided → High
summary: - [regression] Cosmic daily image 20180607 installs but never boots (stuck
- in grub).
+ [regression] Cosmic daily image 20180607 installs but then never boots
+ (stuck in grub).
Daniel van Vugt (vanvugt) wrote :

Here's a diff between my current best known working image and the broken image.

description: updated
Daniel van Vugt (vanvugt) wrote :

The only thing that stands out between the working and broken images to me is:

< ii ubuntu-desktop 1.420 amd64 The Ubuntu desktop system
---
> ii ubuntu-desktop 1.421 amd64 The Ubuntu desktop system
1683c1684
< ii ubuntu-minimal 1.420 amd64 Minimal core of Ubuntu
---
> ii ubuntu-minimal 1.421 amd64 Minimal core of Ubuntu
1692c1693
< ii ubuntu-standard 1.420 amd64 The Ubuntu standard system
---
> ii ubuntu-standard 1.421 amd64 The Ubuntu standard system

Changed in grub-installer (Ubuntu):
importance: High → Critical
Changed in grub2 (Ubuntu):
importance: High → Critical
Changed in ubuntu-meta (Ubuntu):
importance: Undecided → Critical
summary: - [regression] Cosmic daily image 20180607 installs but then never boots
+ [regression] Cosmic daily images 20180606-8 install but then never boot
(stuck in grub).
description: updated

Is this legit? Just a mistake?

ubuntu-meta (1.421) cosmic; urgency=medium

  * Refreshed dependencies
  * Moved open-iscsi to server-recommends (LP: #1630946)

 -- root <root@c.lxd> Wed, 30 May 2018 07:01:29 +0000

What daily image are we talking about?

How does "stuck in grub" present? Do you get thrown into a grub prompt, or do you just see nothing after the grub menu disappears?

Changed in grub2 (Ubuntu):
status: New → Incomplete
Daniel van Vugt (vanvugt) wrote :

We are talking about "cosmic daily images" http://cdimages.ubuntu.com/daily-live/

It's just the grub prompt.

Changed in grub2 (Ubuntu):
status: Incomplete → New
Jean-Baptiste Lallement (jibel) wrote :

I cannot reproduce this issue in a VM. Could you provide more details about the machines you tried to install on, is it UEFI hardware or with a specific setup?

Daniel van Vugt (vanvugt) wrote :

It's seemingly got nothing to do with the hardware.

Today I have tested cosmic image 20180611 on two machines:

  * Lenovo X1 Carbon (gen 5)
  * Dell XPS 13 (9343)

and the bug persists. In both cases Ubuntu installs fine, but then the machine never boots.

I'm lucky in that I still have image 20180530 as a workaround.

Any chance I can get access to images 20180531-20180605 to better bisect the bug?

description: updated
summary: - [regression] Cosmic daily images 20180606-8 install but then never boot
+ [regression] Cosmic daily images 20180606-11 install but then never boot
(stuck in grub).

This is probably coincidence but the working images are slightly bigger than the non-working ones:

-rw------- 1 dan dan 1964244992 May 24 10:46 cosmic-desktop-amd64-20180522.iso
-rw------- 1 dan dan 1964244992 May 26 14:53 cosmic-desktop-amd64-20180525.iso
-rw------- 1 dan dan 1964244992 May 30 15:48 cosmic-desktop-amd64-20180530.iso
-rw------- 1 dan dan 1957183488 Jun 8 12:56 cosmic-desktop-amd64-20180606.iso
-rw------- 1 dan dan 1957183488 Jun 7 15:55 cosmic-desktop-amd64-20180607.iso
-rw------- 1 dan dan 1957183488 Jun 8 15:51 cosmic-desktop-amd64-20180608.iso
-rw------- 1 dan dan 1957183488 Jun 13 12:40 cosmic-desktop-amd64-20180611.iso

Łukasz Zemczak (sil2100) wrote :

As already asked by Jean-Baptiste, could you give us more info on whether the installation has been done in UEFI or legacy BIOS mode? Is it broken for both cases? We did have a new grub2 released quite recently which might have potentially some effect here - but I am also unable to reproduce the issue when installing on a VM (tried ubuntu-server though). Let me check ubuntu-desktop maybe.

Daniel van Vugt (vanvugt) wrote :

I am always using UEFI.

I don't think grub2 is the problem. This bug pre-dates that update by about 1 day. And image 20180606, which contains the bug, does not contain the new grub2:

ii grub-common 2.02-2ubuntu8 amd64 GRand Unified Bootloader (common files)
ii grub-gfxpayload-lists 0.7 amd64 GRUB gfxpayload blacklist
ii grub-pc 2.02-2ubuntu8 amd64 GRand Unified Bootloader, version 2 (PC/BIOS version)
ii grub-pc-bin 2.02-2ubuntu8 amd64 GRand Unified Bootloader, version 2 (PC/BIOS binaries)
ii grub2-common 2.02-2ubuntu8 amd64 GRand Unified Bootloader (common files for version 2)

See comment #2.

It's reproducible in a VM with a Cosmic desktop image and UEFI enabled.

no longer affects: ubuntu-meta (Ubuntu)
Changed in grub2 (Ubuntu):
status: New → Confirmed
no longer affects: grub-installer (Ubuntu)
affects: grub2 (Ubuntu) → ubuntu-meta (Ubuntu)

Installer log on a UEFI VM

affects: ubuntu-meta (Ubuntu) → grub2 (Ubuntu)
Changed in grub2 (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
summary: - [regression] Cosmic daily images 20180606-11 install but then never boot
- (stuck in grub).
+ [regression] Cosmic daily images 20180606-11 install but boots to grub
+ prompt on EFI systems

It seems the subiquity images are not affected. I have first tried the latest cosmic ubuntu-server-live image and was able to boot correctly into the system after installation. Then I have tried the old debian-installer based ubuntu-server images and was able to reproduce the issue, with the system displaying the grub prompt. This explains why I was not able to reproduce the issue before (I was using subiquity). Both on UEFI.

summary: - [regression] Cosmic daily images 20180606-11 install but boots to grub
- prompt on EFI systems
+ [regression] Cosmic daily images 20180606-11 install but boots only to
+ grub prompt on EFI systems
Changed in grub2 (Ubuntu):
assignee: Canonical Foundations Team (canonical-foundations) → Łukasz Zemczak (sil2100)
status: Confirmed → In Progress

For now I'm just writing down some of my random thoughts, but looking at Daniel's diff between the working image and non-working, I don't see anything straightforward that could have caused it. That being said, Dimitri had a good idea that one of the livecd-rootfs changes could have caused it. I now see this which might be somehow related?

https://launchpad.net/ubuntu/+source/livecd-rootfs/2.528

Time-wise it could fit, as the package - even though uploaded on the 28th of May, it actually only migrated from cosmic-proposed to cosmic on the 5th of June (our images are built without -proposed enabled). Need to look into the change itself, but it does deal with grub somehow.

Daniel van Vugt (vanvugt) wrote :

Yes, 'livecd-rootfs' is also a candidate since it doesn't appear in 'dpkg -l' so is hidden from my diff.

Changed in linux-meta (Ubuntu):
importance: Undecided → Critical
Changed in livecd-rootfs (Ubuntu):
importance: Undecided → Critical
affects: linux-meta (Ubuntu) → ubuntu-meta (Ubuntu)
Daniel van Vugt (vanvugt) wrote :

Comment #9 also suggests the problem might have arisen from a change to 'livecd-rootfs'

Łukasz Zemczak (sil2100) wrote :

I am starting to doubt the theory that livecd-rootfs was responsible. I have built the livefs locally using livecd-rootfs 2.537 (so pre-dating the broken images), repacked the latest cosmic iso to include my squashfs in it, installed ubuntu-server on UEFI and am still experiencing the same grub prompt bug. Either my test-case is invalid or the issue lies somewhere else?

Łukasz Zemczak (sil2100) wrote :

Looking at the change in livecd-rootfs it seems to be indeed not related, as the modified code-path is only executed for ubuntu-cpc projects.

Writing down some of our thoughts. Me and Jean-Baptiste sat down on the issue just now and noticed a strange thing happening in the ESP. We had 3 installs: A) cosmic ubuntu-server (debian-installer), B) cosmic ubuntu-server-live (subiquity) system and C) bionic ubuntu. A) is obviously broken, we get the grub2 prompt after booting. All cases but A) work, but even for A) it's possible still to boot the system by directly loading the config from the main partition.

We compared the ESP contents in A), B) and C). For B) the grub.cfg file (along with the .efi binaries) is located in the EFI/ubuntu/ directory on the efi partition. For C) there are 2 grub.cfg files present (and the corresponding binaries), one in EFI/ubuntu/ and one in EFI/grub/. Both of those systems boot. However, for A) the ESP only has the EFI/grub/ part. It looks like something that usually installed the required bits to ubuntu/ has changed and now, possibly, this is causing grub not proceeding with its boot process. Problem is we can't seem to see any changes in any of the installer packages that could have touched these parts.

Łukasz Zemczak (sil2100) wrote :

It looks like the grub2-signed upload is the culprit, looks like the dependency switch we made caused in the d-i cases for grub2-efi-amd64 not to be installed, even though it's required. Will be consulting a fix.

I think it's because grub-efi-amd64 is missing. Installing it in /target at the end of installation (after remounting the efi partition) fixes the problem.

Changed in grub2 (Ubuntu):
status: In Progress → Triaged
no longer affects: livecd-rootfs (Ubuntu)
Changed in grub2-signed (Ubuntu):
importance: Undecided → Critical
status: New → Triaged
Łukasz Zemczak (sil2100) wrote :

Since the grub2-signed changes that have introduced this regression are in fact needed, we will be looking into fixing this in the installer.

no longer affects: ubuntu-meta (Ubuntu)
Changed in ubiquity (Ubuntu):
status: New → In Progress
Changed in debian-installer (Ubuntu):
status: New → In Progress
importance: Undecided → Critical
Changed in ubiquity (Ubuntu):
importance: Undecided → Critical
Changed in debian-installer (Ubuntu):
assignee: nobody → Łukasz Zemczak (sil2100)
Changed in ubiquity (Ubuntu):
assignee: nobody → Łukasz Zemczak (sil2100)
Łukasz Zemczak (sil2100) wrote :

I would need someone to take a look and review my proposed fix for debian-installer. I have tested that on a custom ISO image and as a result I got my system booting. It did cause the grub prompt to appear (asking on which partition to install), but I suppose that's not anything new.

Changed in grub2-signed (Ubuntu):
status: Triaged → Won't Fix
Changed in grub2 (Ubuntu):
status: Triaged → Won't Fix

@sil2100, Colin already commented on a similar change you proposed against ubiquity https://code.launchpad.net/~sil2100/ubiquity/+git/ubiquity/+merge/348201/comments/909076

tags: added: patch
Changed in grub2 (Ubuntu):
assignee: Łukasz Zemczak (sil2100) → nobody
no longer affects: debian-installer (Ubuntu)
Changed in grub-installer (Ubuntu):
status: New → Fix Committed
importance: Undecided → Critical
assignee: nobody → Łukasz Zemczak (sil2100)
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub-installer - 1.128ubuntu9

---------------
grub-installer (1.128ubuntu9) cosmic; urgency=medium

  * grub-installer: install grub-pc for EFI setups and make sure we're
    not purging it earlier in case it got installed. It's needed to make sure
    the right maintainer scripts are run and the ESP populated. (LP: #1775743)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 19 Jun 2018 10:25:14 +0200

Changed in grub-installer (Ubuntu):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubiquity - 18.10.4

---------------
ubiquity (18.10.4) cosmic; urgency=medium

  [ Chen-Han Hsiao (Stanley) ]
  * Add efivars to mountpoints loaded at bootloader install time.
    (LP: #1772374)

  [ Steve Langasek ]
  * scripts/plugininstall.py: don't hard-code a resume partition in
    /etc/initramfs-tools/conf.d/resume at install time. In bionic and later,
    initramfs-tools will autodetect an appropriate resume partition at
    initramfs generation time, so ubiquity's resume setting is redundant and
    possibly wrong. LP: #1768230.

  [ Łukasz 'sil2100' Zemczak ]
  * Make sure that grub-pc is not removed after installation for both EFI and
    legacy BIOS cases as we now technically need it even for EFI installs.
    (LP: #1775743)

  [ Mathieu Trudel-Lapierre ]
  * Automatic update of included source packages: grub-installer
    1.128ubuntu9.

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 25 Jun 2018 13:57:40 -0400

Changed in ubiquity (Ubuntu):
status: In Progress → Fix Released
Fred Palmer (oldfred) wrote :

Not sure proposed fix is all of issue.

I boot 18.04 by default with UEFI and every test or experimental install in the past has overwritten my /EFI/ubuntu.
But install of Cosmic did not overwrite my /EFI/ubuntu but created /EFI/grub. It has a correct /EFI/grub/grub.cfg, but my UEFI is set to default boot using /EFI/ubuntu.
But booting grub UEFI entry booted me into my default install.
Issue is similar to previous bug reports where a change to GRUB_DISTRIBUTOR= creates a new UEFI entry, but internally either efi_distributor or perhaps just the code that defines where to find grub is hard coded to /EFI/ubuntu/grub.cfg.
It needs to use grub.cfg in the ESP folder it creates. And kluge on Kubuntu should finally be fixed.

Older version, not sure what you have now.

./.pc/install_arm64_naming.patch/util/grub-install.c: efi_distributor = "ubuntu";
./debian/patches/install_efi_ubuntu_flavours.patch:+ efi_distributor = "ubuntu";
./ChangeLog: GRUB_DISTRIBUTOR on efibootmgr line.
./debian/default/grub:GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
./debian/postinst.in: bootloader_id="$(config_item GRUB_DISTRIBUTOR | tr A-Z a-z | \
./debian/changelog: - Adjust UEFI installation to cope with Kubuntu setting GRUB_DISTRIBUTOR

On Tue, Jun 26, 2018 at 09:31:24PM -0000, Fred Palmer wrote:
> But install of Cosmic did not overwrite my /EFI/ubuntu but created
> /EFI/grub. It has a correct /EFI/grub/grub.cfg, but my UEFI is set to
> default boot using /EFI/ubuntu.

With which daily image did you see this?

AFAICS, no image has been released with both fixes yet:
  http://cdimages.ubuntu.com/daily-live/

summary: - [regression] Cosmic daily images 20180606-11 install but boots only to
+ [regression] Cosmic daily images 20180606-11 install but boot only to
grub prompt on EFI systems
Fred Palmer (oldfred) wrote :

> With which daily image did you see this?

The install of Cosmic has modified date of Mon 11 Jun 2018 07∶55∶53 AM CDT.
Not sure when I downloaded it.

Daniel van Vugt (vanvugt) wrote :

AFAIK, all June images older than 20180627 will have the bug.

Please wait until the fix for this bug is released before retesting.

Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1775743

tags: added: iso-testing
Fred Palmer (oldfred) wrote :

Just saw one thread in Ubuntu Forums & one in AskUbuntu where with 18.04 the Boot-Repair output from efibootmgr -v shows no ubuntu entry and only has a grub entry.

See:
https://ubuntuforums.org/showthread.php?t=2396042
And:
https://askubuntu.com/questions/1053895/system-always-boots-to-grub-prompt?noredirect=1#comment1722812_1053895
This user already reported that copying /EFI/grub/grub.cfg to /EFI/ubuntu/grub.cfg fixed his boot issue.

Sean Clarke (belsean21) wrote :

From above comment

https://ubuntuforums.org/showthread.php?t=2396042

Installed from latest Ubuntu Studio 18.04 ISO.

Boots to grub> prompt

Also fixed by copying /boot/efi/EFI/grub/grub.cfg to /boot/efi/EFI/ubuntu/

Daniel van Vugt (vanvugt) wrote :

This bug has been fixed for a while already and is about cosmic only (Ubuntu 18.10).

If you have any problems with 18.04 then please open a new bug.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers