Installation of Ubuntu Groovy with manual partitioning without an EFI System Partition fails on 'grub-install /dev/sda' even on non-UEFI systems

Bug #1893964 reported by Jan Rathmann
122
This bug affects 25 people
Affects Status Importance Assigned to Milestone
ubiquity (Ubuntu)
Fix Released
High
Julian Andres Klode
Hirsute
Fix Released
High
Julian Andres Klode

Bug Description

Hello,

trying to install current daily-live images of Groovy in VirtualBox fails for me when I'm using manual partitioning without an EFI System Partition (ESP).

Steps to reproduce:

1. Partition layout I have used in VirtualBox:
 * Partition table: MBR
 * A single primary ext4 partition (/dev/sda1) using up the entire virtual harddisk with 1 MB free space before start of the partition
2. Boot current Groovy daily-live image, click on "Install Ubuntu"
3. Choose "Something else" (manual partitioning)
4. Select /dev/sda1 as target for '/', check "format partition".
5. Ignore warning about missing EFI system partition.

Result:

The installation proceeds until GRUB is about to be installed.
Then an error dialog appears: "Executing 'grub-install /dev/sda' failed. This is a fatal error." (screenshot attached).
If I click 'Ok', after a moment Ubiquity nevertheless shows the usual dialog saying "Installation is complete. Please restart." The only option the dialog offered was to click on "Restart now".
After that, booting the failed installation succeeds, but it is obvious that Ubiquity couldn't complete its job: Packages like ubiquity itself, which usually get purged from the fresh system at the end of a successful installation, are still installed. There is also a pop-up in gnome-shell showing an error regarding package management (screenshot attached, not sure if this is related to the failed install).

Modifying the setup by creating an ESP manually like described in #7 makes the installation complete successfully without error.

The setups I have tested (my machine and VirtualBox) don't support UEFI or don't have UEFI support enabled respectively (and thus don't actually require an ESP to boot).

To send this report, I was running 'sudo ubuntu-bug ubiquity' on the "failed", but nevertheless booting fresh installation.

Kind regards, Jan

ProblemType: Bug
DistroRelease: Ubuntu 20.10
Package: ubiquity 20.10.9
ProcVersionSignature: Ubuntu 5.4.0-42.46-generic 5.4.44
Uname: Linux 5.4.0-42-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu45
Architecture: amd64
CasperMD5CheckResult: skip
Date: Wed Sep 2 16:55:14 2020
InstallCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-42-generic root=UUID=6ba06971-16c7-4df6-afcc-3bc101cba9a5 ro quiet splash
SourcePackage: ubiquity
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Revision history for this message
Jan Rathmann (kaiserclaudius) wrote :
Revision history for this message
Jan Rathmann (kaiserclaudius) wrote :
Revision history for this message
Jan Rathmann (kaiserclaudius) wrote :

This happens also when I try to install Kubuntu Groovy, the only thing that's different is that after the "grub-install /dev/sda failed" dialog there is for 1-2 seconds a window telling "Sorry the installer had crashed" which disappears again without interventention, and then there is dialog with the following message:

"The installer encountered an unrecoverable error. A desktop session will now be run so that you may investigate the problem or try installing again."

That dialog sends you to the live session.

But that seems to be the only difference between Kubuntu and Ubuntu - the installation will boot despite the failure, but is "not complete", as described above.

Revision history for this message
Jan Rathmann (kaiserclaudius) wrote :

Still happening with Kubuntu daily-live image 20200930.

Revision history for this message
Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1893964

tags: added: iso-testing
Changed in ubiquity (Ubuntu):
milestone: none → ubuntu-20.10
Revision history for this message
Jan Rathmann (kaiserclaudius) wrote :

To be a bit more precise, I found out that this bug isn't limited to VirtualBox, it also happens the same way when I try to install Groovy directly on my hardware (older mainboard without UEFI, no ESP partition, LVM setup, Btrfs filesystem on root).

summary: - Installation of Ubuntu Groovy in VirtualBox with manual partitioning
- fails on 'grub-install /dev/sda'
+ Installation of Ubuntu Groovy with manual partitioning fails on 'grub-
+ install /dev/sda'
Revision history for this message
Jan Rathmann (kaiserclaudius) wrote : Re: Installation of Ubuntu Groovy with manual partitioning fails on 'grub-install /dev/sda'

Further testing, I found out that the grub installation failure seems to be related to whether there is an EFI System Partition (ESP) or not.

I did the following change to my VirtualBox setup to manually create an ESP:
- Shrink the single partition on my virtual HD with GParted so I get 512MB free space at the end of the disk
- Create an additional primary FAT32 partition with 512MB size, mark it as ESP (with cfdisk)

So I have the following partition layout:
/dev/sda1: primary partion, size ~9,5GB, ext4
/dev/sda2: primary partition, size 512MB, fat32, marked as ESP

With this partition layout and selecting /dev/sda1 as root (marked for formating) the installation will complete successfully without an error regarding grub installation, and the installed system boots fine.

So it seems that this bug happens if there is no ESP on the system - even if the system doesn't support UEFI at all and thus doesn't need an ESP.

summary: - Installation of Ubuntu Groovy with manual partitioning fails on 'grub-
- install /dev/sda'
+ Installation of Ubuntu Groovy with manual partitioning without an EFI
+ System Partition fails on 'grub-install /dev/sda' even on non-UEFI
+ systems
description: updated
Revision history for this message
Jan Rathmann (kaiserclaudius) wrote :

Manual workaround for those who are affected:

1. Start Ubiquity by running 'ubiquity -b' (doesn't install grub and will not trigger this bug)
2. After installation is completed, don't reboot and install grub manually with the following commands in a terminal (assuming your have installed Ubuntu to /dev/sda1, change accordingly):

sudo mkdir uroot
sudo mount /dev/sda1 uroot
sudo mount -o bind /dev uroot/dev
sudo mount -o bind /sys uroot/sys
sudo mount -t proc /proc uroot/proc
sudo chroot uroot
grub-install /dev/sda1
update-grub
exit
sudo umount uroot/proc
sudo umount uroot/sys
sudo umount uroot/dev
sudo umount uroot

3. Then reboot, new installation should boot up fine.

Revision history for this message
Jan Rathmann (kaiserclaudius) wrote :

Sorry, typo in my last post, the command to install grub of course has to be:

grub-install /dev/sda

tags: added: rls-gg-incoming
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubiquity (Ubuntu):
status: New → Confirmed
Revision history for this message
ajgreeny (ajg-charlbury) wrote :

I have this same problem when installing Xubuntu-20.10 using KVM/QEMU and virt-manager.
The iso file boots properly and live system runs as expected but there is a cpmplete failure to install grub to /dev/sda.
Looking at the filesystem in thunar shows that this appears to be a UEFI install with an EFI partition and also for some reason a small BIOS-boot partition, but attempting to install grub to either of those two partitions also fails leaving the installed system unbootable.
A bug was created by the installer after this failure at https://bugs.launchpad.net/bugs/1899370

Changed in ubiquity (Ubuntu):
milestone: ubuntu-20.10 → none
tags: added: rls-hh-incoming
removed: rls-gg-incoming
Revision history for this message
Brian Murray (brian-murray) wrote :

Given that there is a warning presented for this situation we've decided to same this for fixing in the HH release of Ubuntu.

Revision history for this message
Derk Willem te Bokkel (dtebokkel) wrote :
Revision history for this message
nmaxx (nmaxx) wrote :

Where is that warning mentioned in comment #12 ?
I did not find anything in the 20.10 release notes' "known issues" list: https://discourse.ubuntu.com/t/groovy-gorilla-release-notes/15533
And do I read this correctly that you'll not release a fixed 20.10 image but wait for 21.04 ?

Revision history for this message
Jan Rathmann (kaiserclaudius) wrote :

@nmaxx I think Brian's comment is not referring to the release notes but to the warning during the installation process where the installer complaints that installation will likely fail without an ESP.

Revision history for this message
Steve Langasek (vorlon) wrote :

Two guidelines we should enforce here:

1. we shouldn't allow the user to leave the partitioner with a config that will lead to an install failure later
2. if the user can't have an ESP for whatever reason, we should let the install proceed without setting up shim-signed/grub-efi*

We should still encourage wherever possible that the user set up an ESP, for compatibility with firmware changes or moving disks between hosts.

Changed in ubiquity (Ubuntu):
importance: Undecided → High
tags: removed: rls-hh-incoming
tags: added: fr-876
Revision history for this message
Benno Schulenberg (bennoschulenberg) wrote :

Same error (grub-install failing) when trying to install Ubuntu Mate onto bare metal. I have no ESP (and have no room to make one), but saw no warning about a missing ESP.

What's worse: the installation failed to boot: it hung while trying to mount something. Rebooting into another distro and examining the /etc/fstab on the Mate partition, it contained an entry for /boot/efi on /dev/sdb3 -- the installation medium! Surely this should never happen: any entries in /etc/fstab that refer to the installation medium should be removed before rebooting, also when any amount of errors occur during installation. After removing the line, I could then boot into Mate.

Revision history for this message
Vitaliy Kanev (vitalkanev) wrote :

HP 15 (ac-101ur model, just in case) laptop owner here. Booted Ubuntu Groovy thumb drive using Legacy method, *no* UEFI whatsoever.

When I first tried to install Ubuntu, replacing Windows (as I now own a new laptop), Ubiquity yelled "No UEFI partition found" - this bug didn't occur in earlier versions of Ubuntu (Bionic and Disco, as tested)

When I *manually* converted hard drive's partition table to 'msdos' in GParted, and trying to manually partition the system, I also get the same "No UEFI partition found" error.

After discarding this error, installation decided to fail at "grub-install" failed. When I looked at the installation console, I did not actually see any bad signs of GRUB failure at all - it just said "Installation complete without error" and installer decided to fail. I don't currently have logs

I am in process of testing workaround posted in Comment #8 and Comment #9. I'll update this comment (or create a new one) if something useful happens.

Thanks.

Revision history for this message
Vitaliy Kanev (vitalkanev) wrote :

Update on the above message: Thanks ~kaiserclaudius for the workaround. It worked :)

In case you are wondering, laptop *does* indeed support UEFI, but I rather prefer using Legacy boot because can go wrong at any moment.

Revision history for this message
Jeremie Tamburini (jeremie2) wrote :

Same problem on 21.04, Ubuntu installer thinks that BIOS is UEFI.

Just 2 months left before the the final release. Any chance for a fix?

Revision history for this message
ChrisOfBristol (chrisofbristol-deactivatedaccount) wrote :

Same here. Ubuntu Budgie 20.10 not installable, 21.04 not installable. This problem makes this OS unusable for me (and probably many other ordinary users) as I won't take the risk of using commands (suggested above) that I don't understand perfectly. I have often found a typo or other error in such commands which cause the commands to fail and the consequences at this stage could be dire. See post #9 for an example of this.

Changed in ubiquity (Ubuntu Hirsute):
assignee: nobody → Julian Andres Klode (juliank)
status: Confirmed → In Progress
Changed in ubiquity (Ubuntu Hirsute):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubiquity - 21.04.15

---------------
ubiquity (21.04.15) hirsute; urgency=medium

  * scripts/plugininstall.py: Only install grub to ESP if ESP is configured
    (LP: #1893964)

 -- Julian Andres Klode <email address hidden> Wed, 14 Apr 2021 17:29:57 +0200

Changed in ubiquity (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Revision history for this message
Jeremie Tamburini (jeremie2) wrote :

Hi,
I've tried to install the 16/4 daily build with Ubiquity 21.04.15, but I'm still experiencing the same problems.

I've just in case filed this bug report: bug #1924823

I've tried to manually install Ubuntu on one of the Ext4 primary partition.
Didn't try any other modality, just seen that selecting the "install on the entire disk" option a popup window tells that an ESR partition will be created.

Revision history for this message
Jeremie Tamburini (jeremie2) wrote :

There's an easier WORKAROUND in case you have no space for an extra EFI partition.

First of all a resume of the issues related to 'EFI on BIOS'.
The system itself get installed. Problem is that Ubiquity thinks that an ESR (EFI) partition is needed even on computers with BIOS. And it does installs grub, but on the USB install device. This because it finds an EFI partition in the USB device and automatically selects it bug #1924823

It's still possible to boot the installed system just keeping the USB device connected to computer.
The WORKAROUND works like this: boot the system, modified /etc/fstab, remove grub-efi* package and re-installed grub-pc.

HERE ALL THE STEPS:

1. Make sure 'first boot from hard disk' is set in the BIOS settings.
2. Connect the USB installer device and start the computer.

3. Open /etc/fstab and comment the line related to /boot/efi:
sudo -H gedit /etc/fstab

4. find a line like this:
UUID=BF7E1-08E7 /boot/efi vfat umask=0077 0 1

5. put the character # at the beginning of the line:
#UUID=BF7E1-08E7 /boot/efi vfat umask=0077 0 1

6. Save the file and exit from gedit.

7. Un-mount e mount the partitions:
sudo umount -a && sudo mount -a

8. Uninstall all grub-efi related packages:
sudo apt purge grub-efi*

9. Reinstall grub-pc:
sudo apt install --reinstall grub-pc

Revision history for this message
Liam Proven (lproven) wrote :

In replie to @jeremie2 in comment #24:

I don't think this is a general description of the problem, because for me, my USB boot keys don't have separate EFI boot partitions.

Revision history for this message
Jeremie Tamburini (jeremie2) wrote :

@lproven
Good observation.

The EFI boot partition is created inside USB device at least with the latest versions of major gui utilities available in Ubuntu: gnome-disk-utility and usb-creator-gtk.
balenaEtcher also does it.

Revision history for this message
Liam Proven (lproven) wrote :

@jeremie2

Ah, fair enough. Mostly I use Ventoy these days, and once the USB key is formatted with Ventoy, you just copy .ISO files onto it and they automagically appear in the Ventoy boot menu. So no need for Balena Etcher etc. any more. Ventoy itself is bootable on BIOS and UEFI PCs and on Intel Macs.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Jeremie, you reported a separate bug for the issue of the installer automatically picking up the USB stick's USB, please comment there if you have to.

Revision history for this message
Jeremie Tamburini (jeremie2) wrote :

> juliank:
> Jeremie, you reported a separate bug for the issue of the installer automatically picking
> up the USB stick's USB, please comment there if you have to.

I know and I apologize, but here is where many other users experienced problems related to EFI/BIOS and I thought it could be a good thing to let them know of this alternative workaround, surely less intimidating than chroot.

Revision history for this message
Sai Vinoba (saivinob) wrote :

I can confirm that the fix is working as on 20210418 iso (as tested on Ubuntu and Xubuntu).

Notes:
1) We still see 'No EFI system partition found..." pop-up which may confuse users.
2) Letting installer create partition on empty disk creates a GPT partition layout with bios_grub, efi and root, or if we choose install alongside option, installer creates an EFI partition.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Hi Sai!

That's the correct behaviour, we do want to create the ESP, and we also do want you to create an ESP when partitioning.

The messaging is not entirely accurate on BIOS systems, as absence of ESP is not immediately fatal, but only would be when you switch to UEFI booting.

I think keeping the message as is was the correct choice for this release compared to adding a new text that's completely untranslated.

Revision history for this message
Liam Proven (lproven) wrote (last edit ):

Just to bump this... I am still getting a warning about a missing ESP when installing 22.04 on BIOS-based systems.

I get a warning that the system has no ESP and probably will not boot. Of course, on BIOS, I don't _need_ an ESP and it boots fine.

For clarity as to *why*:

One of the systems I am testing on contains (right now) Win10, FreeBSD, Ubuntu, Mint, Pop_OS, Devuan, openSUSE Leap, and Garuda.

Because it has a BIOS, only 4 primary partitions are allowed. That is the maximum.

Windows insists on 2 MBR partitions, a Windows Reserved volume and a C: drive. I removed any recovery partitions. FreeBSD can only install into a primary partition; it cannot run from a secondary partition at all.

So I only have 1 left. This is my Extended partition with all the Linux distros in it. I literally have no room for an ESP. I have 4 MBR partitions and that is all that is possible.

Note, none of Devuan, Debian, openSUSE, Manjaro, EndeavourOS, Garuda, MX Linux, or any other non-Ubuntu-based distro complained.

Pop_OS, Linux Mint, Ubuntu Cinnamon and Ubuntu Unity all complained.

This problem is unique to Ubuntu and does not affect Debian or Debian derivatives, as far as I can tell.

Revision history for this message
Julian Andres Klode (juliank) wrote :

@Liam Please read the previous comment - This is a feature, not a bug.

Revision history for this message
Jeremie Tamburini (jeremie2) wrote :

Please reconsider this feature.
It's just creating problems for users who wish to install Ubuntu on computer with old bios.

Last example on our forum just a few days ago: https://forum.ubuntu-it.org/viewtopic.php?p=5306206#p5306206

It's even a wrong message... a missing EFI partition on those computers doesn't prevent a successful boot or installation.
In many cases (like Liam's case) there isn't actually any available primary partition for the EFI.

Cheers

Revision history for this message
Julian Andres Klode (juliank) wrote :

Improving messages is 1983719

Primary partitions not being available is pretty much irrelevant here. You'll have to get a warning either way, your system may still be using UEFI, or you might move the disk to a UEFI system and it would stop working, so you need to be notified about that.

Revision history for this message
Liam Proven (lproven) wrote :

I *have* read the previous comments, all of them.

This is *not* a "feature". This worked correctly in all Ubuntu versions up until 19.10.

The correct behaviour is very simple:
* detect if system booted using UEFI or BIOS
* if UEFI, require ESP
* if _not_ UEFI, do not check for ESP, do not create one, do not warn, install GRUB to MBR or root FS and continue.

Again: no other distro makes this mistake.

I review Linux distros professionally and have done since the 1990s.

For what it is worth I have now criticised this behaviour twice in published articles to millions of readers and I will continue to do so until it is fixed.

The argument about potentially moving a distro to from a BIOS machine to a UEFI one is bogus; the OS won't boot without GRUB in the MBR which is incorrect for UEFI and won't work anyway on a UEFI machine.

It doesn't work the other way either: I have tried moving a UEFI install to a BIOS machine and that will not boot, and reinstalling GRUB did not work.

This is a mistake introduced in Ubuntu 19.10 by someone who did not consider the impact on BIOS-based systems. Fedora works and does not complain, as do Debian, Alpine and Arch.

This is a BUG and it is unique to Ubuntu post Eoan.

Revision history for this message
Julian Andres Klode (juliank) wrote (last edit ):

No this is a feature. Both bootloaders are setup. You can boot an install in both modes, and toggle CSM on and off as much as you want and Ubuntu continues booting.

Users should not end up with an unbootable machine after disabling CSM just because they had CSM enabled when the installer was booted, and then reset their firmware or disabled it.

We considered the impact on BIOS systems, but the benefits far outweigh the minor cosmetic issues.

Revision history for this message
Jeremie Tamburini (jeremie2) wrote :

@juliank
OK you won, it looks things are working :)

When all the troubles came out with the 20.10 release a did tried all the possible combinations with partitions and I remember I had problems with the EFI in a logical partition. I did a new test and this time it was successful.

I managed to grab an old computer with BIOS and I have:
- installed Ubuntu 22.04 manually creating the EFI in a logical partition in a disk with msdos (MBR) partition table
- moved the hard disk to a computer with UEFI
- booted the system selecting the hard drive from the EFI menu
...and the system booted regularly.

Now the major problem is more related to the content of the message that doesn't describe properly the purpose of the EFI on old fashion computers:
Julian Andres Klode has already opened a new bug report:
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1983719

P.S.
For most users setting the EFI in disk with MBR doesn't make sense. This is quite a big feature and a big change, it should have been communicated/documented in a better way.
Ubuntu has all the channels to make its users aware ;)

Revision history for this message
Liam Proven (lproven) wrote :

> No this is a feature. Both bootloaders are setup.

I understand the argument now but I am not convinced. I still think that the real situation here is not being understood.

*If* this is a clean install on an empty hard disk, then OK, argument accepted.

Additionally: *if* one of the automatic disk partitioning options is accepted, either "use the whole disk" *or* the option to automatically partition a disk and take some space off Windows, OK.

But there is a circumstance which is not being considered here: if the user picks "do something else" *and* ALL of the following set of conditions are met:

 - the PC is in BIOS mode, which means no ESP is needed;

 - AND the disk is MBR, which means that only 4 partitions are allowed and requiring an unnecessary additional partition is actively harmful;

 - AND there is an existing OS, meaning the ability to change firmware boot modes is already limited by the other, which means you CANNOT predict what the other OS will do or what it needs.

The error is wrong: no ESP is needed in BIOS mode.

ALSO it is spurious: many PCs can't be toggled, AND if there is another OS, such as Windows or DOS or anything non-Linux, then it MUST NOT be toggled because it will break the other OSes.

AND it is wasting a precious resource.

Whoever had this idea did not think it through enough, and the alleged "feature" of being able to boot in 2 modes does not apply if the machine is dual-booting.

Basically if it is a custom install, it is in BIOS mode, and there's another OS, then it means the user cannot, may not and must not change the boot mode, then DON'T COMPLAIN.

The errror is spurious, it gives false info, it pointlessly alarms users about something that is not a problem.

If you let users do their own partitioning then trust them and don't warn them over something that is not a problem.

The design is wrong. The decision is bogus. The assumptions are incorrect.

This a Red Hat level of wrong-headedness, and I speak as a former RH employee.

It is against the Ubuntu guiding principles of freedom and helping users.

It is actively user-hostile because of things that whoever designed this didn't think of.

Revision history for this message
roland aus köln (devzero-c) wrote :
Revision history for this message
roland aus köln (devzero-c) wrote :

what's even worse - after installing ubuntu besides windows10 on my dell inspiron E5530, i'm getting "invalid partition table" at early boot.

there is efi partition automatically created at /dev/sda5 and linux/ubuntu partition at /dev/sda6 when i choose to install ubuntu besides win10

after reinstall and manually selecting root partition (i.e. use primary partition /dev/sda4 and format with ext4 for rootfs) , the partition error maessage is gone.

so, the ubunutu installer

1. installs an efi partition i did not ask for
2. installs a partition which has no purpose on a legacy booted system
3. is unnecessarily wasting diskspace
4. causing an error, we would not have if it simply wouldn't do stupid things

this sucks

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.