CentOS 8 UEFI image is unbootable

[Michal Nasiadka]

Diskimage-builder built images for CentOS 8 end up in grub shell when booted via UEFI.

After package-installs run with grub2 element, CentOS is not UEFI bootable.

New grub2-efi-* packages put grubenv as a link to /boot/efi/EFI/centos, and
if previous grubenv file exists - it won't move it, just leave the link as
grubenv.rpmnew. This leads to unbootable system via UEFI.

It seems that after CVE-2020-10713 (famous BootHole) some changes were introduced, and now grubenv needs to be linked to /boot/efi/EFI/centos/ - without that grub shell can't boot.

[Ian Wienand (iwienand)]

OK, unfortunately I can't replicate this. I am building basically the smallest image

 DIB_RELEASE=8 disk-image-create vm centos-minimal block-device-efi

and then I import that into virt-manager, switch the firmware type to EFI and boot it. It works for me from master. So I'm not sure what exactly is wrong.

The important grub bits I'm seeing installing are

---
2020-09-01 04:43:08.136 | > Installing : grub2-common-1:2.02-87.el8_2.noarch 11/107
2020-09-01 04:43:08.136 | > Installing : grub2-tools-minimal-1:2.02-87.el8_2.x86_64 20/107
2020-09-01 04:43:52.715 | > Installing : grub2-tools-1:2.02-87.el8_2.x86_64 69/107
2020-09-01 04:44:06.214 | Installing : grub2-tools-extra-1:2.02-87.el8_2.x86_64 3/8
2020-09-01 04:44:06.219 | Installing : grub2-pc-modules-1:2.02-87.el8_2.noarch 4/8
2020-09-01 04:44:06.266 | Installing : grub2-efi-x64-1:2.02-87.el8_2.x86_64 6/8
2020-09-01 04:44:06.399 | Installing : grub2-pc-1:2.02-87.el8_2.x86_64 7/8
2020-09-01 04:44:06.407 | Installing : grub2-efi-x64-modules-1:2.02-87.el8_2.noarch 8/8
---

I think this has the CVE changes? How are you building the images?

[Michal Nasiadka (mnasiadka)]

Well, it seems centos-minimal element doesn't produce the issue, but centos does.

[Ian Wienand (iwienand)]

I can't replicate this with the centos element either.

I'm using

---
DIB_RELEASE=8 disk-image-create vm centos block-device-efi
---

then importing that with virt-manager, switching boot type to EFI and booting. I see the grub menu and it boots correctly.

---
2020-09-06 23:29:23.452 | Installing : efi-filesystem-3-2.el8.noarch 1/3
2020-09-06 23:29:23.452 | Installing : grub2-efi-x64-1:2.02-87.el8_2.x86_64 2/3
2020-09-06 23:29:23.621 | Installing : grub2-efi-x64-modules-1:2.02-87.el8_2.noarch 3/3
---

I think we need more details on what versions are being used, and possibly what other elements might be getting in the way (though not many i think would interfere with the bootloader).

The other thing is the build environment; we've seen leaks between the build platform, where we've got /sys, /proc etc mapped from, and building the images. I think we've turned on all the "ignore what the host system says" flags for grub, dracut etc. but something might still be breaking. I'm building on a very plain ubuntu bionic image.

[Ian Wienand (iwienand)]

Interesting; building RHEL from image, I do see:

---
2020-09-07 01:01:08.246 | grub2-tools-1:2.02-87.el8_2.x86_64 marked as user installed.
2020-09-07 01:01:08.247 | grub2-pc-1:2.02-87.el8_2.x86_64 marked as user installed.
2020-09-07 01:01:08.247 | grub2-efi-x64-1:2.02-87.el8_2.x86_64 marked as user installed.
2020-09-07 01:01:08.247 | grub2-efi-x64-modules-1:2.02-87.el8_2.noarch marked as user installed.
2020-09-07 01:01:08.423 | trying grub2-install
2020-09-07 01:01:08.427 | Installing GRUB2...
2020-09-07 01:01:08.435 | Installing for i386-pc platform.
2020-09-07 01:01:08.481 | /usr/sbin/grub2-install: error: cannot rename the file /boot/grub2/grubenv.new to /boot/grub2/grubenv: No such file or directory.
---

[Ian Wienand (iwienand)]

I'm trying to understand what is going on

https://bugzilla.redhat.com/show_bug.cgi?id=1497918

talks about a similar problem that relates to trying to install efi grub on a non-efi machine.

When you strace it, the problem is

---
/usr/sbin/grub2-install: error: cannot rename the file /boot/grub2/grubenv.new to /boot/grub2/grubenv: No such file or directory

19264 rename("/boot/grub2/grubenv.new", "/boot/grub2/../efi/EFI/redhat/grubenv") = -1 ENOENT (No such file or directory)
---

this is with

fbfbee4e3ba3bd8e56b44595b91e9e14cbd155dd24bd3efd3448995a1e6bb5e2 rhel-8.2-update-2-x86_64-kvm.qcow2

[Ian Wienand (iwienand)]

here's the difference I think

grubenv is a file in the centos image; it's a symlink in the RHEL one

---
root@ianw-dib-builder:~/.cache/image-create# tar tvf CentOS-8-GenericCloud-8.2.2004-20200611.2.x86_64.qcow2.tgz | grep grubenv
-rw-r--r-- root/root 1024 2020-06-11 02:39 ./boot/grub2/grubenv
root@ianw-dib-builder:~/.cache/image-create# tar tvf rhel-8.2-update-2-x86_64-kvm.qcow2.tgz | grep grubenv
lrwxrwxrwx root/root 0 2020-07-28 21:08 ./boot/grub2/grubenv -> ../efi/EFI/redhat/grubenv
---

[Ian Wienand (iwienand)]

I have pulled apart centos8.1, centos8.2, rhel8.1 & rhel8.2 .qcow2 images, and it seems the rhel8.2 image has changed to have EFI partitions, and consequently the symlink is left dangling in that case.

http://paste.openstack.org/show/797556/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/750279

[Ian Wienand (iwienand)]

I've been trying to replicate this, but can't at the moment. I've updated [1] with comments.

The RHEL bug [2] seems similar; however I can't replicate *that* either. And moreso, I can't even get that to build with two issues [3,4]

I'm not sure what else to do

[1] https://review.opendev.org/#/c/748157/
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1871669
[3] https://review.opendev.org/749773
[4] https://review.opendev.org/750279

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750279
Committed: https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=8f12d9530ed79359fb988688caadfa6dc318f7a5
Submitter: Zuul
Branch: master

commit 8f12d9530ed79359fb988688caadfa6dc318f7a5
Author: Ian Wienand <email address hidden>
Date: Tue Sep 8 16:46:29 2020 +1000

    bootloader: remove dangling grubenv links

    The rhel8.2 .qcow2 images have moved from a single xfs partition into
    an EFI style layout with separate /boot/efi partition.

    This means the root partition has /boot/grub2/grubenv as a symlink to
    ../efi/EFI/redhat/grubenv, which is dangling when we are running the
    bootloader element as /boot/efi is blank. grub-install tries to call
    rename() in the process of re-writing this file, which fails and bails
    out dib.

    Remove this symlink if it exists

    Change-Id: I5591144a3617dbae148b0c5d2a6a404942ffcd4e
    Parital-Bug: #1893029
    Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1871669

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michal Nasiadka (<email address hidden>) on branch: master
Review: https://review.opendev.org/748157