ESP on install medium automatically configured as ESP for install, failing install

[Jeremie Tamburini]

This bug report is a duplicated of bug #1902269 performed with Ubuntu 20.10.
Basically installation fail when the installer doesn't find an EFI (ESR) partition on computers with legacy BIOS.

This time I have used Ubuntu 21.04 also to test the new version of Ubiquity package (21.04.15) which, as reported here: https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1893964/comments/22 should have fixed the issue.

Unfortunately I'm still experiencing the same problem.

#### Partition details:
- msdos partition table
- 4 primary partitions, all ext4

---
Apr 17 00:18:54 ubuntu grub-installer: info: Identified partition label for /dev/sda4: msdos
Apr 17 00:18:54 ubuntu grub-installer: info: Installing grub on '/dev/sda'
Apr 17 00:18:54 ubuntu grub-installer: info: grub-install does not support --no-floppy
Apr 17 00:18:54 ubuntu grub-installer: info: Running chroot /target grub-install --force --target x86_64-efi "/dev/sda"
Apr 17 00:18:54 ubuntu grub-installer: Installazione per la piattaforma x86_64-efi.
Apr 17 00:19:07 ubuntu grub-installer: grub-install: errore: impossibile copiare "/usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed" su "/boot/efi/EFI/ubuntu/grubx64.efi": Spazio esaurito sul device.
Apr 17 00:19:07 ubuntu grub-installer: error: Running 'grub-install --force --target x86_64-efi "/dev/sda"' failed.
---

ProblemType: Bug
DistroRelease: Ubuntu 21.04
Package: ubiquity 21.04.15
ProcVersionSignature: Ubuntu 5.11.0-14.15-generic 5.11.12
Uname: Linux 5.11.0-14-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu65
Architecture: amd64
CasperMD5CheckResult: pass
CasperVersion: 1.461
Date: Sat Apr 17 02:20:20 2021
InstallCmdLine: BOOT_IMAGE=/casper/vmlinuz file=/cdrom/preseed/ubuntu.seed maybe-ubiquity quiet splash ---
LiveMediaBuild: Ubuntu 21.04 "Hirsute Hippo" - Beta amd64 (20210416)
RebootRequiredPkgs:
 linux-image-5.11.0-14-generic
 linux-base
SourcePackage: ubiquity
UpgradeStatus: No upgrade log present (probably fresh install)

[Chris Guiver (guiverc)]

Also related (hirsute)
- https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1907180
- https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1923134
- https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1923699

Note: on 1923134 I did the following

ubuntu@ubuntu:/$ df -h |grep target
/dev/sda1 658G 328G 297G 53% /target
/dev/sdb2 4.9M 4.9M 2.0K 100% /target/boot/efi

so it looks like grub is being written to the installation media (not sda) as per pasted messages from ubiquity syslog

[Julian Andres Klode (juliank)]

You must have accidentally mounted something into (/target)/boot/efi, as otherwise the code does not trigger anymore. It seems likely you did infact mount a removable ESP into it.

[Julian Andres Klode (juliank)]

Hmm I did not get the warning message about missing ESP when booting the ISO as an usb stick instead of a CD. This is odd.

[Julian Andres Klode (juliank)]

So what happens is that partman automatically discovers the ESP on the USB stick, and configures it as EFI system partition (you can see it says efi in the line). You have to go to the partition and tell it to not use it.

Not sure what the right fix is. Cheap hack would be to only include EFI system partitions that are big enough, which would have the side effect of fixing this 5MB partition issue.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1924823

[Jeremie Tamburini (jeremie2)]

Hi and thanks for your reply.
I'm going to do some more tests.

Anyway, if you select "Erase the disk and install Ubuntu" the installer is going to create an ESP on /dev/sda (see the attached picture). I didn't perform this type of installation (I did it few months ago and it actually create the ESP partition) I just wanted to see if the installer was still doing this.

It looks like it doesn't matter what your are doing, the installer is "acting" like the only existing system is UEFI.

[Chris Guiver (guiverc)]

Ubuntu hirsute QA-test on install on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)

manual partitioning, created new new partition table
sda1 will be ext4 using all available space
(ISO written to media via mkusb/dus; live; but I've had same result before writing ISO with `dd bs=4M of..`)

failed with this issue... on box (ctrl+alt+T to open term)

ubuntu@ubuntu:/$ mount |grep target
/dev/sda1 on /target type ext4 (rw,relatime,errors=remount-ro)
/dev/sdb2 on /target/boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/sdb1 on /target/cdrom type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048)
sysfs on /target/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
udev on /target/dev type devtmpfs (rw,nosuid,noexec,relatime,size=2430848k,nr_inodes=607712,mode=755,inode64)
proc on /target/proc type proc (rw,relatime)
tmpfs on /target/run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=498164k,mode=755,inode64)

the box only has single HDD which is sda... installer said it was installing boot loader to sda, so I can confirm @Jeremie2's results on current daily

[Jeremie Tamburini (jeremie2)]

Some more info:

- I repeated the installation selecting the "don't use this partition" option for the EFI partition of the USB device.
The result is still the same like in the example of guiverc:
/dev/sda1 on /target type ext4
/dev/sdb2 on /target/boot/efi

- I filed a new bug report: bug #1924856
I thought it might be useful to have some data when installation succeed using one of the automatic option. In this case I've selected "Install Ubuntu alongside other system".
An ESR partition is automatically created on /dev/sda3.

Till Ubuntu 20.04 everything was fine. There still were two different modality to boot the USB/dvd device: "uefi modality" and "old bios modality".
All these issues started with Ubuntu 20.10 where the way to boot the USB/dvd device is now unified.

[Chris Guiver (guiverc)]

Further to comment #9 (with the idea of work-around) I initial-booted the thumb-drive (installation media) but by editing the grub entry (so instead of booting it normally; it booted the installed (sda1) system instead using `chainloader (hd1)+1`)

and my installed version booted.

I ejected the thumb-drive and tried to grub-install but get

guiverc@dc7700:~$ sudo grub-install /dev/sda
[sudo] password for guiverc:
Installing for i386-pc platform.
grub-install: warning: this GPT partition label contains no BIOS Boot Partition; embedding won't be possible.
grub-install: warning: Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and their use is discouraged..
grub-install: error: will not proceed with blocklists.

I'm fearful this won't boot... so no workaround..

[Chris Guiver (guiverc)]

comment #11 didn't boot :(

[Jeremie Tamburini (jeremie2)]

Got the WORKAROUND and it's quite easy :D
(Thanks guiverc, you gave the idea!)

As you have done I tried to boot the system keeping the USB installer device connected to the computer. I've just set 'first boot from hard disk' inside bios settings... and the system boots.
Then I've uninstalled grub-efi* package, modified /etc/fstab and re-installed grub-pc. It worked!

HERE ALL THE STEPS:

1. Make sure 'first boot from hard disk' is set in the BIOS settings.
2. Connect the USB installer device and start the computer.

3. Open /etc/fstab and comment the line related to /boot/efi:
sudo -H gedit /etc/fstab

4. find a line like this:
UUID=BF7E1-08E7 /boot/efi vfat umask=0077 0 1

5. put the character # at the beginning of the line:
#UUID=BF7E1-08E7 /boot/efi vfat umask=0077 0 1

6. Save the file and exit from gedit.

7. Un-mount e mount the partitions:
sudo umount -a && sudo mount -a

8. Uninstall all grub-efi related packages:
sudo apt purge grub-efi*

9. Reinstall grub-pc:
sudo apt install --reinstall grub-pc

[Julian Andres Klode (juliank)]

Comment #11 - it looks you installed on GPT without a bios boot partition, so there's no space to embed BIOS grub. The installer will have warned you, please listen to it.

Jeremie, the correct workaround is to deselect the EFI system partition when you configure partitions in the installer, as I explained in comment #5.

[Jeremie Tamburini (jeremie2)]

> juliank
> Jeremie, the correct workaround is to deselect the EFI system partition when
> you configure partitions in the installer, as I explained in comment #5.

Let's say that the EFI partition is not selected, it only shows the type (EFI) but it's not selected a mount point.
As reported in comment #10 I have also tried the installation explicitly selecting "don't use this partition" for the EFI on the USB stick. But it didn't work, same problems of the other attempts.

[Chris Guiver (guiverc)]

@juliank, the install was done on a nearby test box (near my primary system) simulating a usual install which really will be a MBR/legacy partition table

the GPT was automatically created when I selected new (I'd have chosen MBR if given a choice); the systems where this matters are all MBR and have all four primary partitions already utilized

No warning messages were shown I'm sure !. http://iso.qa.ubuntu.com/qatracker/milestones/419/builds/229694/testcases/1302/results is the QA-test report.

If a message had shown, I'd likely have realized it was GPT & gone back and created MBR (or pre-prepared as I'd done in most prior QA-test installs in hirsute/this cycle using gparted) as that is my actual intended install case (ie. upgrade an existing/prior Ubuntu install from long ago).

The simple 'install' strategy was used ONLY to make it easier to re-produce the problem.

Reports made duplicate of this include my actual install boxes with systems I do care about (d755-5, d780 etc). All are multi-boot with all primary partitions already utilized.

[Chris Guiver (guiverc)]

@juliank & (differs to my comment #16)

Using the latest daily (so different ISO) I repeated the steps (comment #9) and do get an warning today (I don't recall it before, but maybe I'm mistaken)

---
"The partition table format in use on your disks normally requires you to create a separate partition for boot loader code. This partition should be marked for us as a "Reserved BIOS boot area" and should be at least 1MB in size. Note that this is not the same as a partition mounted on /boot.

If you do not go back to the partitioning menu and correct this error, boot loader installation may fail later, although it may still be possible to install the boot loader to a partition"
---

Again I don't believe I got this error; but it's there today... as @juliank mentioned in comment #14

[Chris Guiver (guiverc)]

I prepared dc7700 using MBR/dos partition table & single partition (done using gparted on hiruste live).

Did a QA-test install equivalent to comment #9 except partitioning was done using gparted before ubiquity was run.

(MBR/dos partition table, single partition being sda1)
(representing my many boxes that have MBR/dos partition tables & four primary partitions, and an upgrade-via-re-install; thus no-format was selected on sda1. I did get warning about old files impacting install, but partition was empty (valid warning given how I'm using it though)

grub-install failed & system failed to boot.
no warning messages as per my last comment #17
for comments written as I QA-test refer to http://iso.qa.ubuntu.com/qatracker/milestones/419/builds/229742/testcases/1302/results/

alas I forgot to `mount |grep target` as I've done some of my prior installs, but on screen all reported boot loader was going to sda (disk) not sdb (installation-media-thumb-drive)

[Julian Andres Klode (juliank)]

I have applied a workaround in ubiquity that makes it ignore ESPs smaller than 50 MB, which should fix this bug for most users, unless some popular tool reformats the USB stick itself and creates a larger ESP.

[Chris Guiver (guiverc)]

@juliank I saw your update last night, and hoped I'd see a new RC/daily to test your applied workaround, so if there something I can help test.. let me know..

(I've had this issue on 2x dell optiplex 755, a 780 & hp dc7700 but I suspect it occurs regardless of bios box.. the dc7700 & one d755 I use only in QA-testing; the others I actually use so are only rarely used for install testing (& always replace a partition type installs)... I've returned to lubuntu QA-testing)

[Jean-Baptiste Lallement (jibel)]

ubiquity (21.04.18) hirsute; urgency=medium

  * Only auto-configure ESP larger than 50 MiB - this works around LP 1924823,
    where the 5 MB ESP on the install medium was configured as /boot/efi.

[Chris Guiver (guiverc)]

I just noticed #21 and have the prior daily running with an updated `ubiquity`... and noticed the following on drawing of the ubiquity dialog.

It appears some text is drawn but then overwritten by the sda1 map (green bar on the picture)... If this is the message that @juliank says I should have seen, I'd not noticed it before, but I believe I noticed text written just prior the the sda1 green line appearing..

This could be the reason why @juliank says to note a message, but it's not appearing clearly on the screen at all (I'd not noticed it before; it was only there a flash as the screen drew)

It appears to have a yellow (or red?, orange? brown?) circle before the line of text - but I can only guess.

Refer https://photos.app.goo.gl/JWwQmn314JzC9fzVA

You might notice edges of it are slightly visible underneath the green bar IF you study it... I'd not noticed it prior installs (only saw it this time as it appeared to momentarily appear before the green bar)

[Julian Andres Klode (juliank)]

@jibel The bug was explicitly not closed in the changelog, because this is not the correct fix for it, just a workaround for standard install mediums which have 5 MB size.

[Chris Guiver (guiverc)]

The important bit started in #22.

By booting the older daily (sudo apt update; sudo apt full-upgrade) to get the fixed `ubiquity` & other packages... then starting the install..

The install I'd done before
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
manual partitioning, pre-prepared, DOS/mbr/legacy with single sda1

install completed; no seen errors..
rebooted & I can login to ubuntu hirsute :)

Thanks @juliank

(not recorded as QA-test as not pure daily/RC)

[Jeremie Tamburini (jeremie2)]

Hi everybody,
As guiverc I have updated Ubiquity on a older Ubuntu iso image to get version 21.04.19

In good part I can confirm good news!
...but there are still bizarre things

I've performed 2 test installation:

1- MANUALLY CHOSING A PARTITION
Same result reported by guiverc at comment #24
The bad thing is the message 'No EFI Partition found. This system will likely not be able to boot... please go back and add an EFI...'
This can be tricky!

2- INSTALL ALONGSIDE... (in remaining disk space - 3 partition already taken on MBR)
That's a typical situation seen many times on old computers with Windows7 that already takes 3 partitions.
It finally works! But it's quite weird.
During the installation the well known grub error message "Executing 'grub-install /dev/sda' failed This is a fatal error" appears.
But rebooting the system everything is fine. I'm actually writhing from the installed system.
I'm afraid that the error message can be even trickier than the message in the previous case.

I've filed bug #1925187 before rebooting the system not knowing if the system could boot. Then I've changed title and description... anyway, log files are there for further information.

I presume there is no much time before the release dayleft for a 'cleaner' solution. It's a pity that only now someone is doing something against these issues.
Anyway... juliank and also jibel, thank you very much for all the efforts.

P.S. I'll probably repeat the tests when the new daily-build is out.

[Jeremie Tamburini (jeremie2)]

Just to confirm everything I've written in message #25.
I've repeated the same tests with the deily iso 20/4.

The workaround at message #13 is no more needed for Ubuntu 21.04, but it can still be useful for Ubutnu 20.10

The step:
sudo apt install --reinstall grub-pc

can be more easily replaced by:
sudo update-grub

[Chris Guiver (guiverc)]

Ubuntu Desktop hirsute RC, performed tests on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
- dell [optiplex] 755 (c2d-e8300, 8gb, amd/ati radeon rv610/radeon hd2400 pro/xt)

listed on http://iso.qa.ubuntu.com/qatracker/milestones/419/builds/229822/testcases/1302/results/

without issue. System installed, on reboot I had Ubuntu hirsute.

FYI: The message that quickly appears then gets covered by the 'space map' was unreadable on both boxes, with only 2-4 words being read by myself before it was covered, and that was with me waiting for it & trying to quickly read it both times.

The message I don't believe actually fitted on screen anyway (on the 1024x768 display) but it wasn't there long enough to be readable anyway so it doesn't matter (I can't be sure of what I saw at all as it felt 'subliminal')

[Brian Murray (brian-murray)]

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.