Ubuntu doesn't provide \EFI\BOOT\BOOTX64.EFI for UEFI systems

Bug #1366546 reported by Benjamin Tegge on 2014-09-07
76
This bug affects 14 people
Affects Status Importance Assigned to Milestone
grub2 (Ubuntu)
Undecided
Unassigned
shim (Ubuntu)
Undecided
Unassigned
ubiquity (Ubuntu)
Undecided
Unassigned

Bug Description

Ubuntu (desktop/server platform AMD64) as of 14.04 just installs shim.efi and grubx64.efi in \EFI\ubuntu\ on the EFI System Partition (ESP). This is unreliable as more and more laptops are introduced to the market where manufacturers choose to reduce (cripple) the firmware functionality to boot only from \EFI\BOOT\BOOTX64.EFI. To make the situation worse legacy boot is also *not implementent* anymore in the firmware. Mostly mainstream consumer oriented and low budget devices are affected.

Ubuntu should backup contents of \EFI\BOOT\ (these are just normal files on FAT32, zipping or taring the directory should be suffcient, another run of the installer however should detect an existing backup of the original and not overwrite the backup of the original) and install a UEFI loader to \EFI\BOOT\BOOTX64.EFI that can boot at least Windows and Ubuntu.

My suggestion would be to use gummiboot [1] in conjunction with PreLoader+HashTool [2] as this will allow functional UEFI Secure Boot (preloader will provide keys to shim to close the gap in gummiboot) and being of low complexity or low resource footprint.

Contents to be installed to the ESP:

\EFI\BOOT\BOOTX64.EFI
> This is a renamed PreLoader.efi.
\EFI\BOOT\HashTool.efi
\EFI\BOOT\loader.efi
> This is a renamed gummiboot.efi to be easily picked up by HashTool (loader.efi is the default).

content of gummiboot configuration file: \loader\entries\ubuntu-grub-shim.conf
    title Ubuntu GRUB (Secure Boot)
    efi \EFI\ubuntu\shimx64.efi

content of gummiboot configuration file: \loader\entries\ubuntu-grub.conf
    title Ubuntu GRUB
    efi \EFI\ubuntu\grubx64.efi

content of gummiboot configuration file: \loader\loader.conf
    default Ubuntu GRUB (Secure Boot)
    timeout 4

If UEFI Secure Boot is enabled, PreLoader will run HashTool on first boot to enroll the hash of gummiboot. The TUI dialogs are simple and straight forward, but the user should be informed that this might happen when installation on a Secure Boot enabled target has been completed. Please have a look the ASCII art representation of the TUI dialogs [3].

Notes:
- gummiboot is not currently packaged in Ubuntu or Debian AFAIK. While having gummiboot packaged as a full GRUB replacement [4] would be nice, we need only the loader binary (gummiboot.efi) to fix this bug.
- An existing Windows UEFI loader on the ESP is automatically detected and doesn't need to be configured.
- Replacing an exsiting \EFI\BOOT\BOOTX64.EFI is similar to overwriting the MBR, it's nasty but the most simple solution for the corresponding design.

I need to emphasize this:
- Legacy boot is not available on these machines.
- Users with affected laptops don't understand the issue they run into. -> They can't install/boot Ubuntu anymore, while no other OS vendor did anything particular evil (till now).
  - Giving users instructions which requires little knowledge of how to use the terminal is too difficult to follow for them (they find it nearly impossible and not very ubuntu-like)
  - The community has a lot of users who don't know the difference between booting with UEFI or legacy BIOS, thus suggesting false troubleshooting or tools that don't work or add more confusion to the situation.
  - Telling users to buy fully functional or non-crippled hardware has always been difficult.
- This a major issue that should be fixed with the next LTS 14.04.x release update or probably an emeregency update of the live media. Waiting more than one year may result in Ubuntu becoming less and less attractive to novice users.

I'm looking forward to work with you on this issue and provide further information where possible.

[1]: http://freedesktop.org/wiki/Software/gummiboot/
[2]: http://blog.hansenpartnership.com/linux-foundation-secure-boot-system-released/
[3]: http://askubuntu.com/a/520351/40581
[4]: http://www.freedesktop.org/wiki/Specifications/BootLoaderSpec/

Phillip Susi (psusi) wrote :

The UEFI standard states that BOOT\BOOTX64.EFI is only for use on removable media and that an OS installing to an internal disk is supposed to put its boot loader in its own directory, and the firmware must boot said loader when instructed to via EFI variables.

If there are broken systems out there that don't follow the standard, we would need more details on which ones, how common they are, and confirmation that they are indeed broken, and do not obey the efi boot variables.

affects: ubiquity (Ubuntu) → grub2 (Ubuntu)
Changed in grub2 (Ubuntu):
status: New → Incomplete
Benjamin Tegge (livewirebt) wrote :

Thank you for your reply. I tested a Toshiba Satellite NB10-A-10N as a user reported having trouble with a device of that model series (NB10t-A-101, the "t" in the NB10-A series seems to indicate touch capability). I just looked into the UEFI Specification Version 2.4 (Errata B) and found "3.4.1.1 Removable Media Boot Behavior":

> On a removable media device it is not possible for the FilePath to contain a file name, including sub directories. FilePathList[0] is stored in non volatile memory in the platform and cannot possibly be kept in sync with a media that can change at any time. [...]

From my point of view these sentences state: The information which loader to boot from the supported filesystems is stored in the platform (simply speaking, an entry in the NVRAM of the mainboard) and not available when you take out the drive and try to boot it on another computer.

> If FilePathList[0] points to a device that supports the EFI_SIMPLE_FILE_SYSTEM_PROTOCOL, then the system firmware will attempt to boot from a removable media FilePathList[0] by adding a default file name in the form \EFI\BOOT\BOOT{machine type short-name}.EFI.

This describes that booting can be done through \EFI\BOOT\BOOT{arch}.EFI. I think we agree on that, I'm just adding it for completeness.

I admit that I haven't read the entire spec, but I do think that there is a misunderstanding here regarding what removable media is or is not and that this section defines a fallback for usecases when information that is only stored in the platform is not available like installing Ubuntu to an external harddrive, which is still possible with UEFI. (This fallback should be deployed with the default Ubuntu installation as it would enable bootable external and internal installations, no matter if a firmware implementation is considered to be broken, crippled or non-standard.)

I have seen many cases over at AskUbuntu that I would consider being occurrences of this issue in retrospect. We would need a tool (or testcase) that users can run to gather further data if a system follows the standard and boots so called NVRAM entries successfully or requires above method to execute a bootloader in UEFI mode that belongs to Ubuntu.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/17/2014 12:26 AM, Benjamin Tegge wrote:
> This describes that booting can be done through
> \EFI\BOOT\BOOT{arch}.EFI. I think we agree on that, I'm just adding
> it for completeness.

Right... *for removable media*. The normal mechanism is for each boot
loader on the disk to be registered in the efi variable boot catalog,
but this obviously can not be done for removable media so there is a
fallback provided for that case. This fallback is generally only used
when you explicitly press a key during boot to request booting from
something other than the usual boot loader, and it only allows for
there to be a single boot loader on the removable media. For
installations on fixed disks you want to be able to have multiple boot
loaders installed so you can run different operating systems.

> I have seen many cases over at AskUbuntu that I would consider
> being occurrences of this issue in retrospect. We would need a tool
> (or testcase) that users can run to gather further data if a system
> follows the standard and boots so called NVRAM entries successfully
> or requires above method to execute a bootloader in UEFI mode that
> belongs to Ubuntu.

I suspect that such cases are the result of bugs preventing the efi
boot catalog from being properly updated ( there have been several of
those ). The correct solution is to fix the bug and get the boot
catalog updated correctly rather than to violate the efi spec and
blindly overwrite the fallback boot loader.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQEcBAEBAgAGBQJUToaSAAoJEI5FoCIzSKrwo6wH/0JYfO9xWFsknM1C/tUqYu16
lRyYQEI4tRguVD1Q+ceWSjknH9c12x6QhcUyR2g048aFjj9GjJUM8RRJfKmUpgCc
9yhzJcA2pZrcuE/df1HB5PkCp45HLV8PEA8xYj3KQHBezCJ6r289ve9FThnUNhFd
WObV817umg4SJtkTISV7YLYAy+asU6HS+t/ARAx98HM6Mwq/R8xhcj5BhshtDC1K
dis4U+UBOmZinWnK8WQwD2dQoTYg7XXbNrDrnUN/B7YnKtqv9vioVaY5xwu/KMjc
9bdCIyfU1NWAI8PExdmr/7EQdXhTjlgEwMknKILZaanvUUcXpaeagPpiMTifqik=
=+c+e
-----END PGP SIGNATURE-----

Susan Cragin (susancragin) wrote :

This may be the bug that is affecting me.
I have a Lenovo Carbon 2. During about half the kernel updates in Utopic, the computer subsequently refused to boot. AT ALL.
I had the bios setting set to dual, with UEFI first and legacy second. But changing bios setting to "legacy" did not work.
My only solution was to re-install with a different kernel.
(I went back to Trusty, which worked.)
I have another Lenovo computer that did not have this problem at all, runs Utopic like a champ, with every update.
I have been unable to file a bug report because I seem to have no useful information.

I am worried because today, 10/29 Trusty installed kernel 39 during my morning updates. So I ran update-grub immediately to ascertain which kernel I was currently running, and which I would next boot into.
My currently working kernel is 37.
(You will note that I have lowlatency installed too. I have tested. There is no difference in boot behavior between low-latency and generic. )

Found linux image: /boot/vmlinuz-3.13.0-39-lowlatency
Found initrd image: /boot/initrd.img-3.13.0-39-lowlatency
Found linux image: /boot/vmlinuz-3.13.0-39-generic
Found initrd image: /boot/initrd.img-3.13.0-39-generic
Found linux image: /boot/vmlinuz-3.13.0-37-lowlatency
Found initrd image: /boot/initrd.img-3.13.0-37-lowlatency
Found linux image: /boot/vmlinuz-3.13.0-37-generic
Found initrd image: /boot/initrd.img-3.13.0-37-generic
Adding boot menu entry for EFI firmware configuration
done

If I shut my computer down and then re-boot and I cannot boot, which is what I expect will happen, I will update this comment.

Phillip Susi (psusi) wrote :

This would prevent the system booting at all, ever, not only sometimes after an update.

Susan Cragin (susancragin) wrote :

That's what happens. No boot, no nothing. Ever. Black screen. No way to get to a grub menu or command line.
The only way I can get my computer to boot is with a usb key, after going into bios to set it to boot with that.
So that's what I do, then re-install the entire system.
To update:
I re-booted into Trusty. Everything worked.
With foolish optimism, I then upgraded to Utopic.
Re-booted .... nothing at all.
So I reinstalled Trusty.

Phillip Susi (psusi) wrote :

I mean you wouldn't have been able to boot Trusty.

Susan Cragin (susancragin) wrote :

So I've hooked my problem to the wrong bug and should file a new one?

Phillip Susi (psusi) wrote :

Yes.

Fred Palmer (oldfred) wrote :

I have not read the spec but saw this:

UEFI 2.3 -it is permitted for the firmware to treat the EFI system partition in the same way as removable media or use bootx64.efi
http://mjg59.dreamwidth.org/4125.html

And many UEFI implementations, all of HP & Sony that I have seen in Ubuntu forums will only boot "Windows" and usually copying grubx64.efi is one work around that then lets users boot.

Real solution is that vendors do not violate UEFI spec.

Vendors violated UEFI specs - http://hwe.ubuntu.com/docs/ubuntu-bios-uefi-requirements.pdf
[quote]Firmware should not enforce any boot policy other than the mechanism specified in Section 3 of the
UEFI 2.3.1 specification [UEFI 2.3.1]. Specifically, firmware should not modify boot behaviour de-
pending on the Description field of the EFI_LOAD_OPTION descriptor.[/quote]

Phillip Susi (psusi) wrote :

Does Windows install a copy of its boot loader to BOOT\BOOTX64.EFI? I didn't think it did.

Benjamin Tegge (livewirebt) wrote :

Windows 8 does and it seems to be bootmgfw.efi as it looks on my laptop (it has 3 drives, 1 HDD for Windows, 1 HDD for Ubuntu LTS and Fedora, 1 SSD for the latest Ubuntu, all UEFI):

livewire@za21:~$ unset LANG
livewire@za21:~$ mount | grep ^/dev
/dev/sdb3 on / type ext4 (rw,errors=remount-ro)
/dev/sdb1 on /boot/efi type vfat (rw)
/dev/sdd1 on /media/livewire/SANDISK type vfat (rw,nosuid,nodev,uid=1000,gid=1000,shortname=mixed,dmask=0077,utf8=1,showexec,flush,uhelper=udisks2)
/dev/sda1 on /mnt/test type vfat (rw)
/dev/sda2 on /media/livewire/Windows 8 T530 type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
livewire@za21:~$ grep -v microsoft -U /mnt/test/EFI/Boot/bootx64.efi
Binary file /mnt/test/EFI/Boot/bootx64.efi matches
livewire@za21:~$ md5sum /mnt/test/EFI/Boot/bootx64.efi
bad97e7203aec2bd026403a7f70688b9 /mnt/test/EFI/Boot/bootx64.efi
livewire@za21:~$ md5sum /media/livewire/Windows\ 8\ T530/Windows/Boot/EFI/bootmgfw.efi
bad97e7203aec2bd026403a7f70688b9 /media/livewire/Windows 8 T530/Windows/Boot/EFI/bootmgfw.efi

I don't want to screw around with NVRAM on that machine, but I recorded a video on another machine:
https://dl.dropboxusercontent.com/u/15877132/w8_bcdboot_bootx64.mkv

It's only 504,2kB in size (30 fps desktop capture with vlc, no audio and nothing else particularly fancy).

I mounted the ESP with `mountvol b: /s` in Windows, renamed the existing bootx64.efi and ran bcdboot, which created a new ~1.5 MB bootx64.efi-file. Unfortunately, this being another machine (another disk/partition layout or another license/edition) the hashes don't match.

Also my experience from previously mentioned case is that manufacturers put a customized Windows loader in that location which tries to blend POST and Windows boot phase together by displaying the same company logo the whole time (probably to give the impression of a faster startup).

Phillip Susi (psusi) wrote :

Sigh... good going microsloth for violating the uefi spec already.

I see two possibilities:

1) The machine is using the default bootloader because for some reason, ubuntu failed to update the efi boot catlog to point to grub.

2) The machine's buggy firmware assumes that if the default loader exists, it should use that, no matter what the boot catalog says.

If you can run efibootmgr -v on an affected machine, this could show which theory is correct.

Benjamin Tegge (livewirebt) wrote :

Hello Phillip and everyone else,

> ubuntu failed to update the efi boot catlog to point to grub.

I checked on the Toshiba Satellite NB10-A-10N during my testing and after the installation was done when I ran `efibootmgr -v` the typical all lowercase "ubuntu" entry was in the output. When I restored the drive to the initial state and removed the entry to reinstall with secure boot enabled the result was the same on this machine: the catalog was updated. I think we can rule that out.

> 2) The machine's buggy firmware assumes that if the default loader exists, it
> should use that, no matter what the boot catalog says.

I haven't tested this scenario as far a can remember and I don't have access to such a machine currently.

Would it be a good idea to put this up to the top of the report and set a corresponding tag or status so that someone else with such hardware can provide the information?

> good going microsloth for violating the uefi spec already.

I'd rather put the blame on the manufacturers or who ever writes the firmware, but that wouldn't help solving the problem I guess. :)

Phillip Susi (psusi) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/11/2014 11:21 AM, Benjamin Tegge wrote:
> I checked on the Toshiba Satellite NB10-A-10N during my testing
> and after the installation was done when I ran `efibootmgr -v` the
> typical all lowercase "ubuntu" entry was in the output. When I
> restored the drive to the initial state and removed the entry to
> reinstall with secure boot enabled the result was the same on this
> machine: the catalog was updated. I think we can rule that out.

Can I assume it was also set as the default boot choice? i.e.
BootOrder listed its number first? It would be worth checking if
deleting the bootx64.efi got it to obey the boot catalog.

> I'd rather put the blame on the manufacturers or who ever writes
> the firmware, but that wouldn't help solving the problem I guess.
> :)

True; it is ultimately on the motherboard manufacturers, but they can
only get away with this because Microsoft installed a boot loader
where they have no business doing so.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQEcBAEBAgAGBQJUYlRDAAoJEI5FoCIzSKrwwPkH/3BICXyPRQE8XumFH7hWAXl+
IS1CbQa6CDViGlTgzHFwhduEXgHj6IuhK9ARBsT075FSK0OrzDG+Nc0qqNha+sSI
UnGjJiL9opmGD8JcZzoLKw4P8wFJ1TwRWf4PupGw6XEYcKJrrOL7mKwTW6EG99pY
l8FzcMa5VFk0IY/o/ksrCkvPC+qSlTSZUjdgSjsD88e1lHCzcQ6NoQZpQW44lprP
Vw4mr8Oycz28mveVw6EV2uk//uIKzNd0xr3INMyF3kXfi0fLPtW0thbSEhvhNUDy
QpiB6BndKSNMD/fbfhb0mmh9VzWjEpZuzqr51lRX47QLE9j46IEFpczqRXUMuCA=
=hkk+
-----END PGP SIGNATURE-----

Benjamin Tegge (livewirebt) wrote :

Yes it was set as the default first and active item in the BootOrder in efibootmgr -v output. Also worth to mention – if I haven't already – is that the device only offers to change the priority of physical devices in firmware settings, OS specific NVRAM entries are at least not displayed anywhere in the menus.

Launchpad Janitor (janitor) wrote :

[Expired for grub2 (Ubuntu) because there has been no activity for 60 days.]

Changed in grub2 (Ubuntu):
status: Incomplete → Expired
Marco Ciampa (ciampix) wrote :

Please "refresh" this bug report since buggy bios laptop are very common ... see Dell Precision T5810

Changed in grub2 (Ubuntu):
status: Expired → Confirmed
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in shim (Ubuntu):
status: New → Confirmed
Changed in ubiquity (Ubuntu):
status: New → Confirmed
Aurélien (aurelienkun) wrote :

Same here : Asus N550JV laptop.

Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1366546

tags: added: iso-testing

As previously expressed; we probably *shouldn't* ship EFI/Boot/bootx64.efi; but since Windows often does, there might be no way around it.

Firmwares are supposed to follow the BootEntries to know what path to use for booting (ie. we create "ubuntu", which points to \EFI\ubuntu\shimx64.efi). If these entries don't exist, firmware will attempt to use the default paths from \EFI\Boot.

Fortunately, shim comes with a fallback.efi binary. One option to fix this (see [1]) would be to write a bootx64.efi to \EFI\Boot which contains our shim (and other operating systems would do the same), along with fallback.efi. The fallback.efi program will walk through the tree searching for boot.csv files (which is another file we'll need to write) for the details on how to rebuild boot entries, then it will attempt to boot the first entry added.

I haven't tested this at all, and I'm a little bit worried about what might happen for really broken firmware where the entries are still there, but ignored somehow -- I'd like to make sure that fallback.efi does the right thing in this case.

[1] https://blog.uncooperative.org/blog/2014/02/06/the-efi-system-partition/

Changed in ubiquity (Ubuntu):
status: Confirmed → Invalid
Changed in grub2 (Ubuntu):
status: Confirmed → Invalid
Changed in shim (Ubuntu):
status: Confirmed → Triaged

Triaged in shim -- grub and ubiquity should be unaffected.

tags: added: rls-y-incoming
Yuv (yuv) wrote :

Affects also SONY Vaio Pro 13. Experienced this bug in August 2013 when initially staging the brand new machine. Wasted a couple of hours on it. Fixed it manually, essentially doing what Benjamin suggested in the original report. Thought it was a SONY idiosyncrasy not worth reporting.

IT IS SHOCKING TO FIND THIS BUG MENTIONED IN THE RELEASE NOTES TO 16.04LTS (intentional caps).

Sure there is a standard and everybody *should* abide by the standard. However, there are facts on the ground that weigh more and when those facts go against the standard, one has to balance between the cost of flouting the standard and the cost of flouting the user's need.

Cost of flouting the user's need: average users expecting out-of-the-box performance get frustrated with Ubuntu and give up.

Cost of flouting the standard: minimal. So some piece of zealously standard-abiding software will "think" that it was booted from a removable drive? Some extremely well engineered hardware designed for security will disable access to resources, fearing that this is a smuggled removable drive that should not be allowed access to the organization's secrets? All of these are probably unlikely because the same sloppy manufacturer that flouted the standard in the first place would have to implement this kind of safety. And they are quantitatively irrelevant since there are more mainstream consumers trying to install Ubuntu on their devices than highly sophisticated, secured corporations.

Standards are not set in stone. They do change. Keeping this bug alive for so long is one of those papercuts that make ordinary people prefer the proprietary systems. Ordinary people don't care if the system does not abide by the standard. They care that the system works.

Chris Rainey (ckrainey) wrote :

HP 19-2113w AIO desktop:

To fool HP's broken UEFI implementation, put into nvram:

sudo efibootmgr -c -L "Windows Boot Manager" -l "\EFI\Microsoft\Boot\bootmgfw.efi"

Case is not really important in a FAT filesystem. The bootmgfw.efi is the name of the original (now absent) Microsoft bootloader.

Next, set up the bootloaders. Use shimx64.efi as the bootloader. Copy it into <your mount point>/EFI/Microsoft/Boot/bootmgfw.efi (Using the foward slashes you use on a Linux system). Also, copy grubx64.efi into the <your mount point>/EFI/Microsoft/Boot directory (since shim expects it there). I usually leave the grub.cfg file in <your mount point>/EFI/ubuntu/grub.cfg. grubx64.efi will find it there.

That should be a working boot setup for secure or non secure boot . If you don't care about secure boot, you can just copy grubx64.efi into ...bootmgfw.efi

Just as a fallback, you may put another copy of shim (or grubx64.efi) into .../EFI/Boot/bootx64.efi (note name change to bootx64.efi)(and a copy of grubx64.efi there too).

Some machines will try that if the requested boot fails."

THIS WORKS FOR: HP 19-2113w AIO

In fact, I only had to create the .../EFI/Boot/ directory and populate as stated, above. I left the M$ directory completely out(tested multiple times to be sure).

Enter the BIOS, at boot, using F10 and move(drag) the 'ubuntu' entry to the top of the 'Boot Order' list for UEFI. It can get reset during a failed boot!

Also, I'm using the latest BIOS update from hp.com as of 07/13/2015.

SOURCE: http://ubuntuforums.org/showthread.php?t=2256060&page=2

Steve Langasek (vorlon) wrote :

Have discussed with cyphermox on IRC; since the rest of the bootloader installation code is in grub-install, we should do this as well in the same place instead of having grub-install and shim-signed compete with one another.

Changed in grub2 (Ubuntu):
status: Invalid → Triaged
Changed in shim (Ubuntu):
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers