grub efi doesn't install fs module needed to access root

Bug #1652822 reported by Jeremiah Baughman on 2016-12-27
82
This bug affects 14 people
Affects Status Importance Assigned to Milestone
grub2 (Debian)
Fix Released
Unknown
grub2 (Ubuntu)
High
Unassigned
Bionic
Medium
Eric Desrochers

Bug Description

[Impact]

One can't deploy using root XFS partition using 18.04 on UEFI platform.

[Test case]

* Deploy a Bionic machine with a storage layout as follows:

vda-part1 536.9 MB fat32 /boot/efi
vgroot-lvroot 31.7 GB xfs / ## Only xfs is important here.

The Bionic/18.04LTS deployments will fail to boot and will be redirected to grub prompt:
grub>

[Where problems could occur]

I checked the delta between the 2 grub (Bionic and Focal) versions after the add of 'xfs' module to see if thre is any fixes/CVE/... that we should pick up, and I found nothing.

There is this change:
- xfs: Accept filesystem with sparse inodes

But it only seems to affect xfsprogs (4.16.0) which is not found in Bionic.

Ryan, cyphermox and I were concerned that it is adding to the set of modules included in the secureboot-signed binary. I confirmed with vorlon and everything seemed okay on his end.

If something arise, it will be around xfs itself, and shouldn't affect other FS module. Situation would be as it is right now, xfs not working with uefi platform.

[Other information]

* Debian:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911147

debian/changelog:
 - Add xfs module to signed UEFI images (closes: #911147, LP: #1652822).

* Salsa commit:
https://salsa.debian.org/grub-team/grub/-/commit/01f1eee93acd4113475b403f80661cf651fbcc00

* The workaround is to create a /boot partition, with ext4, so you have:

# Example
vda-part1 ext4 /boot
vda-part2 fat32 /boot/efi
vgroot-lvroot xfs /

[Original Description]

If I do a fresh install on an efi system and use xfs for the root and boot filesystems, the install will succeed and then on reboot grub will drop to a grub shell and fail to boot. If I try to do an ls on any of the xfs filesystems, grub reports unsupported filesystem. Doing an insmod on xfs.mod doesn't work on the xfs filesystem since it can't be read

/boot/efi is a fat32 esp partition and can be read by grub. I can make the system boot by putting a copy of /boot/grub/x86_64-efi/xfs.mod in /boot/efi/EFI/ubuntu/xfs.mod and then putting the following line in /boot/efi/EFI/ubuntu/grub.cfg
insmod (hd0,gpt1)/EFI/ubuntu/xfs.mod

I'm guessing this would work if boot was ext2.

My current HDD setup is:
fdisk -l /dev/sda
Disk /dev/sda: 477 GiB, 512110190592 bytes, 1000215216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5F358931-DDAE-4F78-8647-B54434A5A3DC

Device Start End Sectors Size Type
/dev/sda1 2048 2000895 1998848 976M EFI System
/dev/sda2 2000896 4001791 2000896 977M Linux filesystem
/dev/sda3 4001792 804902911 800901120 381.9G Linux LVM

My current fstab is:
/dev/mapper/vgroot-lvroot / xfs defaults 0 0
# /boot was on /dev/sda2 during installation
UUID=006c4d31-8c29-4e25-b8c2-5f70f5f19afd /boot xfs defaults 0 0
# /boot/efi was on /dev/sda1 during installation
UUID=DFD4-FB95 /boot/efi vfat umask=0077 0 1
/dev/mapper/vgroot-lvhome /home xfs defaults 0 0
/dev/mapper/vgroot-lvswap none swap sw 0 0

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: grub-efi-amd64 2.02~beta2-36ubuntu3.2
ProcVersionSignature: Ubuntu 4.4.0-57.78-generic 4.4.35
Uname: Linux 4.4.0-57-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.4
Architecture: amd64
CurrentDesktop: MATE
Date: Tue Dec 27 12:42:40 2016
InstallationDate: Installed on 2016-12-27 (0 days ago)
InstallationMedia: Ubuntu 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
SourcePackage: grub2
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Jeremiah Baughman (stilljb) wrote :
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in grub2 (Ubuntu):
status: New → Confirmed
The Loeki (the-loeki) on 2016-12-28
tags: added: yakkety
Phillip Susi (psusi) on 2017-01-11
summary: - grub efi xfs unsupported filesystem
+ grub efi doesn't install fs module needed to access root
Changed in grub2 (Ubuntu):
importance: Undecided → Critical
Ryan Harper (raharper) wrote :

In analyzing this over in Bug 1731490; grub-install does not check the filesystem of /boot; even if it did, it still needs to do the follwing:

1) copying the proper modules into /boot/efi/EFI/ubuntu/x86_64-efi/<module>
2) updating /boot/efi/EFI/ubuntu/grub.cfg to insmod the required modules for the the search.fs_uuid to locate root/boot

It's not that simple. The module should have been copied if xfs was detected as the filesystem for /boot... But there's more to it than that. In UEFI, you also need a signed grub binary that already includes the module, otherwise Secure Boot might prohibit you loading it from disk.

This is certainly not a Critical bug as it is only as issue for a fraction of our users who happen to be at the intersection of using XFS and UEFI, but at least I think we have enough information to be able to work on the fix.

Changed in grub2 (Ubuntu):
importance: Critical → High
status: Confirmed → Triaged
tags: added: id-5a8df3b7e96d9c9d3ffc8f7b
Brian K. White (brian-aljex) wrote :

Same here. I had to write myself a google drive doc that I could consult every time a the grub-efi package was updated and broke my boot again.

In addition to the steps explained previously, I also copy my working manually edited grub.cfg to emergency.cfg, so that every time I get a failed boot again after a system update, I pull up google drive on my phone to read this emergecy manual boot command:

grub> configfile ${cmdpath}/emergency.cfg

That one command is all you need to enter manually when you are faced with the grub> prompt at boot, and it loads the rest of your normal grub config, and you boot normally.

Then once booted, copy emergency.cfg back over grub.cfg, and update the copy of xfs.mod since it might have been updated by the very update that regnerated grub.cfg.

https://drive.google.com/open?id=1B2FrtIaG75PhFPsjzXnJS0MeDg7rezEP3Ef5fFlrLAo

Changed in grub2 (Debian):
status: Unknown → Fix Released
Diego Remolina (dijuremo) wrote :

This definitively affects 18.04 as well. The main culprit is that the /boot/efi/EFI/ubuntu/grubx64.efi binary in 18.04 lacks XFS support.

Brian K. White (brian-aljex) wrote :
Download full text (5.4 KiB)

On 19.04 I still get my grub.cfg broken by normal updates. Not by kernel updates, and not by running update-grub. But there is some package that whenever that package is updated, part of the update process generates a new grub.cfg, and it's a broken one because it makes a bad assumption somewhere.

I eneded up writing myself a little "check_grub_efi_xfs" script that I run any time I think I saw anything about grub fly by when I run "apt upgrade", or, any time grub gets screwed up I run it after I get myself back in.

It's only about 3 lines of code that actually matters, but writes a lot of stuff to the screen just so you don't have ANY worries about it what it did and if your boot files are currently ok before you shut down.

Ideally this should run just before shutdown on every single shutdown. Then you'd never have to worry about it. The only way you'd miss an update again is if you had an ungraceful shutdown after an apt upgrade that included a grub packege update. But despite being a unix and linux admin for over 20 years, I have so far utterly failed to get systemd to do such a simple task. If someone can show me exactly how to write a freaking unit file that does that, I'd be pretty grateful.

Just to refresh, the issue here is, it's possible to install ubuntu with xfs root fs to a machine that already has a factory Win10 install and boot files in a factory EFI partition, without having to make any special /boot partition. All you really need is to place a copy of xfs.mod onto the EFI partition, which is a FAT filesystem that grub already reads files from at boot. And edit the initial grub.cfg that's in the EFI filesystem so that it loads xfs.mod from there too.

Then everything else is fine. The rest of /boot can be in "/", and "/" can be xfs. This allowed me to degfrag and resize the factory Win10 without wiping and reinstalling, install linux in the freed space, with xfs root fs, and no other partition or filesystem changes needed. I can boot back and forth between the original Win10 install or linux with no problem. It didn't break Win10 booting.

It's a perfectly valid and "legal" arrangement, and anything that automatically writes grub config files should have to allow for this possibility without breaking it.

Screen capture from a run of this fixup script, just after today's apt upgrade which broke my grub.cfg once again.
This is today 2019 02 26, so I don't know where the supposed fix is.
It's definitely not yet in 19.04 or in grub-* 2.02+dfsg1-5ubuntu11

So this run shows the actions when it detects that it actually has something to do.
Most times, it will detect that it doesn't have to do anything, and it will take no action those times.

This, and the script itself (attached) should serve as detailed documentation of the admittedly uncommon configuration, and exactly how it gets broken, and exactly how it gets un-broken.

---snip---
root@negre:~# check_grub_efi_xfs
Checking for broken grub + efi + xfs config...

EFI System Partition ESP_DIR=/boot/efi/EFI/ubuntu
total 3488
-rwx------ 1 root root 108 Aug 14 14:47 BOOTX64.CSV
-rwx------ 1 root root 141 Jan 25 11:13 emergency.cfg
drwx------ 2 root root 4096 Dec 14 ...

Read more...

Brian K. White (brian-aljex) wrote :

Correction to that date I write above: 2019 02 06 not 2019 02 26 blargh

Brian K. White (brian-aljex) wrote :

... Ah I see, it does look like possibly the debian upstream might have fixed it, and merely the ubuntu package has not yet got it. Which is what this bug declares at the top, so no discrepency there after all.

They mention signed images, where in my own case I have uefi boot security disabled and can boot unsigned things, so I don't know if that fix will still fix me or not, but seems likely.

Launchpad Janitor (janitor) wrote :
Download full text (10.1 KiB)

This bug was fixed in the package grub2 - 2.02+dfsg1-12ubuntu1

---------------
grub2 (2.02+dfsg1-12ubuntu1) disco; urgency=medium

  * Merge against Debian unstable; remaining changes (LP: #564853):
    - debian/control: Update Vcs fields for code location on Ubuntu.
    - debian/control: Breaks shim (<< 13).
    - Secure Boot support: use newer patchset from rhboot repo:
      - many linuxefi_* patches added and modified
      - dropped debian/patches/linuxefi_require_shim.patch
      - renamed: debian/patches/no_insmod_on_sb.patch ->
        debian/patches/linuxefi_no_insmod_on_sb.patch
    - debian/patches/install_signed.patch, grub-install-extra-removable.patch:
      - Make sure if we install shim; it should also be exported as the default
        bootloader to install later to a removable path, if we do.
      - Rework grub-install-extra-removable.patch to reverse its logic: in the
        default case, install the bootloader to /EFI/BOOT, unless we're trying
        to install on a removable device, or explicitly telling grub *not* to
        do it.
      - Install a BOOT.CSV for fallback to use.
      - Make sure postinst and templates know about the replacement of
        --force-extra-removable with --no-extra-removable.
    - debian/patches/add-an-auto-nvram-option-to-grub-install.patch: Add the
      --auto-nvram option to grub-install for auto-detecting NVRAM availability
      before attempting NVRAM updates.
    - debian/build-efi-images: provide a new grub EFI image which enforces that
      loaded kernels are signed for Secure Boot: build gsb$arch.efi; which is
      the same as grub$arch.efi minus the 'linux' module. Without fallback to
      'linux' for unsigned loading, this makes it effectively enforce having a
      signed kernel.
    - Verify that the current and newer kernels are signed when grub is
      updated, to make sure people do not accidentally shutdown without a
      signed kernel.
    - debian/default/grub: replace GRUB_HIDDEN_* variables with the less
      confusing GRUB_TIMEOUT_STYLE=hidden.
    - debian/patches/support_initrd-less_boot.patch: Added knobs to allow
      non-initrd boot config.
    - Disable os-prober for ppc64el on the PowerNV platform, to reduce the
      number of entries/clutter from other OSes in Petitboot
    - debian/patches/shorter_version_info.patch: Only show the upstream version
      in menu and console, and hide the package one in a package_version
      variable.
    - debian/patches/skip_text_gfxpayload_where_not_supported.patch: Skip the
      'text' payload if it's not supported but present in gfxpayload, such as
      on EFI systems.
    - debian/patches/bufio_sensible_block_sizes.patch: Don't use arbitrary file
      fizes as block sizes in bufio: this avoids potentially seeking back in
      the files unnecessarily, which may require re-open files that cannot be
      seeked into, such as via TFTP.
    - debian/patches/ofnet-init-structs-in-bootpath-parser.patch: initialize
      structs in bootpath parser.
    - debian/rules: shuffle files around for now to keep build artefacts
      for signing at the same location as they were expected by Launchpad.
    - debian/ru...

Changed in grub2 (Ubuntu):
status: Triaged → Fix Released
Ryan Harper (raharper) wrote :

At least for Xenial, this still fails, the xfs module either not build-in, or not copied.

Eric Desrochers (slashd) on 2021-01-07
Changed in grub2 (Ubuntu Bionic):
status: New → In Progress
assignee: nobody → Eric Desrochers (slashd)
importance: Undecided → Medium
Eric Desrochers (slashd) on 2021-01-07
description: updated
description: updated
description: updated
description: updated
Eric Desrochers (slashd) on 2021-01-07
description: updated
Eric Desrochers (slashd) on 2021-01-07
description: updated
Eric Desrochers (slashd) on 2021-01-07
description: updated
description: updated

Hello Jeremiah, or anyone else affected,

Accepted grub2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02-2ubuntu8.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in grub2 (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Eric Desrochers (slashd) wrote :

I don't fix grub2 often ... And I missed/forgot changing the build-depends version instruction in grub2-signed to no longer point to 8.20 but 8.21.

I just did the modification, re-uploaded, and let sil2100 knows about my oversight

-Build-Depends: debhelper (>= 9), lsb-release, python3, python3-apt, grub-efi-amd64-bin (>= 2.02-2ubuntu8.20) [amd64], grub-efi-arm64-bin (>= 2.02-2ubuntu8.20) [arm64]
+Build-Depends: debhelper (>= 9), lsb-release, python3, python3-apt, grub-efi-amd64-bin (>= 2.02-2ubuntu8.21) [amd64], grub-efi-arm64-bin (>= 2.02-2ubuntu8.21) [arm64]

Sorry for the inconvenient.

- Eric

Eric Desrochers (slashd) wrote :

[VERIFICATION BIONIC]

I have reproduced the problem with grub2 (2.02-2ubuntu8.20) w/ this layout:

/boot/efi
/ xfs

I then boot with a live cd, mount the partitions, mount --bind {dev, proc, run, sys} and chroot to enable proposed and update grub to version 2.02-2ubuntu8.21 and grub2-signed (1.93.24)

And now everything is working fine, and I can no longer reproduce the problem.

- Eric

tags: added: verification-done-bionic
removed: verification-needed-bionic
Eric Desrochers (slashd) wrote :

[PART 2]

Here some output to confirm my above statement:

* release:
$ lsb_release -cs
bionic

* df:
/dev/sda2 24G 8.0G 17G 34% /
/dev/sda1 511M 6.1M 505M 2% /boot/efi

* /etc/fstab:
# / was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/9eb6f661-542a-45e5-adf9-6d93cdd5500d / xfs defaults 0 0
# /boot/efi was on /dev/sda1 during curtin installation
/dev/disk/by-uuid/6966-6E10 /boot/efi vfat defaults 0 0

* proposed pkgs:
dpkg -l | grep -i grub
ii grub-common 2.02-2ubuntu8.21 amd64 GRand Unified Bootloader (common files)
ii grub-efi-amd64 2.02-2ubuntu8.21 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 version)
ii grub-efi-amd64-bin 2.02-2ubuntu8.21 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 binaries)
ii grub-efi-amd64-signed 1.93.24+2.02-2ubuntu8.21 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 version, signed)
ii grub-legacy-ec2 1:1 all Handles update-grub for ec2 instances
ii grub2-common 2.02-2ubuntu8.21 amd64 GRand Unified Bootloader (common files for version 2

Mathew Hodson (mhodson) on 2021-01-17
tags: removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.02-2ubuntu8.21

---------------
grub2 (2.02-2ubuntu8.21) bionic; urgency=medium

  * d/build-efi-images:
    - Add xfs module to signed UEFI images
      (Closes: #911147) (LP: #1652822)

 -- Eric Desrochers <email address hidden> Thu, 07 Jan 2021 08:45:31 -0500

Changed in grub2 (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for grub2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.