Bug #1899993 “EFI: Fails when BootCurrent entry does not exist” : Bugs : curtin package : Ubuntu

Revision history for this message

dann frazier (dannf) wrote on 2020-10-15:

#1

hot-koala-install.log.gz Edit (19.7 KiB, application/octet-stream)

summary:

- Fails when BootCurrent entry does not exist
+ EFI: Fails when BootCurrent entry does not exist

Revision history for this message

Ryan Harper (raharper) wrote on 2020-10-15:

#2

Looking back at your patch:

Index: curtin-20.1-29-g81144052/curtin/util.py
===================================================================
--- curtin-20.1-29-g81144052.orig/curtin/util.py
+++ curtin-20.1-29-g81144052/curtin/util.py
@@ -886,6 +886,7 @@ def get_efibootmgr(target=None):
     """
     with ChrootableTarget(target=target) as in_chroot:
         stdout, _ = in_chroot.subp(['efibootmgr', '-v'], capture=True)
+ LOG.info("DANNF:\n%s", stdout)
         output = parse_efibootmgr(stdout)
         return output

And the logs:

Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmp6ha4_iz2/target', 'efibootmgr', '-v'] with allowed return codes [0] (capture=True)
DANNF:
BootCurrent: 0003
Timeout: 10 seconds
BootOrder: 0003,0004,0005,0006,0001

The firmware appears to have no entries at all; but somehow has a BootCurrent and a BootOrder pointing to entries that don't exist. This certainly has to be a firmware bug no?

Changed in curtin (Ubuntu):
status:	New → Incomplete

Revision history for this message

dann frazier (dannf) wrote on 2020-10-15:

#3

I'm also assuming a firmware bug here, but I need to consult the specification. If it is a clear firmware bug then I think plan A should certainly be to pressure the vendor to release a fix.

Either way, I think we're likely to run into users suffering from this issue for a while. I tend to see issues like this appear on one platform only to reappear on other platforms in the future. And even if there is a firmware solution, we can't expect users to always be running the latest and greatest firmware. I'd therefore propose that we try to detect the issue and warn users of the problem - maybe even linking to this bug.

Revision history for this message

Ryan Harper (raharper) wrote on 2020-10-15:

#4

@Dann

OK. That sounds reasonable. For this issue, curtin can (and I think it makes sense) ignore a BootCurrent value if there is no corresponding entry; warn in the logs that it's missing, check your firmware, etc... and proceed if it can.

Changed in curtin (Ubuntu):
importance:	Undecided → Medium
status:	Incomplete → Confirmed

Revision history for this message

nb (nateybobo) wrote on 2020-10-17:

#5

@dann @ryan

RE: I'm also assuming a firmware bug here, but I need to consult the specification. If it is a clear firmware bug then I think plan A should certainly be to pressure the vendor to release a fix.

Is there a way we can verify that the firmware is the true issue? I have Dell R420's running the latest firmware and can assist with testing if you'd like. If this can't be fixed I'll be forced to completely move away from MAAS and find another solution for bare metal provisioning :(

I posted my issues in Discourse: https://discourse.maas.io/t/maas-is-changing-my-boot-order/3491

Revision history for this message

nb (nateybobo) wrote on 2020-10-18:

#6

@dann @ryan

FYI - I refreshed to Snap 2.9.0~beta6-9039-g.5a2ba747e and it looks like my boot order woes are fixed now.

Revision history for this message

Ryan Harper (raharper) wrote on 2020-10-20:

#7

@nb This issue is *slightly* different than the other bug related to missing BootCurrent value.

The previous bug which is fixed in curtin happens when the BootCurrent value is not present in the efibootmgr output.

This bug has a BootCurrent value, like 0003, however, the Entries provided by efibootmgr do not have one for 0003.

Here's the raw output from efibootmgr -v:

BootCurrent: 0003
Timeout: 10 seconds
BootOrder: 0003,0004,0005,0006,0001

This system has no boot entries; this is a firmware bug.

From the log file, after curtin adds an entry for Ubuntu we can see the boot entries look like this:

{
  "current": "0003",
  "timeout": "10 seconds",
  "order": [
    "0000"
  ],
  "entries": {
    "0000": {
      "name": "ubuntu",
      "path": "HD(1,GPT,0937ffdf-628c-4161-8b2f-5920235669c6,0x800,0x100000)/File(\\EFI\\ubuntu\\shimx64.efi)"
    }
  }
}

Current pointed entry '0003', however, the list of entries only contains '0000' (what curtin just installed). Then when curtin attempts to place the newly installed entry at the beginning of the BootOrder list, it fails when we run this command:

efibootmgr -o 0003,0000
Invalid BootOrder order entry value0003
^
efibootmgr: entry 0003 does not exist

The fix for this in curtin is to verify that BootCurrent points to an existing entry; if not, then we ignore the BootCurrent value.

Revision history for this message

Ryan Harper (raharper) wrote on 2020-10-20:

#8

@Dann

For this firmware bug where we have missing entries; Have you ever booted this up and had efibootmgr show all of the entries?

Looking at this output here:

BootCurrent: 0003
Timeout: 10 seconds
BootOrder: 0000,0003,0004,0005,0006,0001
Boot0000* ubuntu HD(1,GPT,0937ffdf-628c-4161-8b2f-5920235669c6,0x800,0x100000)/File(\EFI\ubuntu\shimx64.efi)

With my suggested change to ignore BootCurrent since it does not have a matching entry; curtin can at best put the new entry 0000 first in order (note this is already done by grub-install).

This does mean that the node will be lost from maas control since it is no longer PXE booting...
Is that acceptable?

Changed in curtin (Ubuntu):
status:	Confirmed → Incomplete

Revision history for this message

dann frazier (dannf) wrote on 2020-10-28:

#9

I tried to prove whether or not this is a firmware issue or, perhaps, a bug in efibootmgr or curtin's calling of it. I deployed an affected system w/ MAAS w/ init=/bin/bash on the cmdline, and was surprised to see that efibootmgr showed the full expected set of Boot variables. However, if I ssh'd in while MAAS was deploying, efibootmgr was missing some. Turns out this depends on which virtual file system efibootmgr is using.

efivarfs is commonly mounted under /sys/firmware/efi/efivars, and is what efibootmgr prefers:

$ ls /sys/firmware/efi/efivars | grep ^Boot
ls: reading directory '/sys/firmware/efi/efivars': Input/output error
Boot mode select-7683c190-9523-4402-81ff-a11e93dc389c
Boot0000-8be4df61-93ca-11d2-aa0d-00e098032b8c
BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
BootOptionSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c

If /sys/firmware/efi/efivars is not there, efibootmgr appears to fallback to /sys/firmware/efi/vars which the kernel always exposes:

$ ls /sys/firmware/efi/vars | grep ^Boot
Boot mode select-7683c190-9523-4402-81ff-a11e93dc389c
Boot0000-8be4df61-93ca-11d2-aa0d-00e098032b8c
Boot0001-8be4df61-93ca-11d2-aa0d-00e098032b8c
Boot0003-8be4df61-93ca-11d2-aa0d-00e098032b8c
Boot0004-8be4df61-93ca-11d2-aa0d-00e098032b8c
Boot0005-8be4df61-93ca-11d2-aa0d-00e098032b8c
Boot0006-8be4df61-93ca-11d2-aa0d-00e098032b8c
BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
BootOptionSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c

With init=/bin/bash, efivarfs is not yet mounted, which is why I see the full set at that point.

So why are these variables not visible in efivarfs? And does the "Input/output error" ls reports have something to do with it?

strace shows that the -EIO is coming from a getdents64() call:

openat(AT_FDCWD, "/sys/firmware/efi/efivars/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
getdents64(3, /* 65 entries */, 32768) = 4800
getdents64(3, 0x5625e2d36510, 32768) = -1 EIO (Input/output error)

I built a 5.10-rc1+ kernel to see if this is maybe a kernel but that was already fixed upstream.
Turns out, the issue goes away there:

$ ls /sys/firmware/efi/efivars | grep ^Boot
Boot mode select-7683c190-9523-4402-81ff-a11e93dc389c
Boot0000-8be4df61-93ca-11d2-aa0d-00e098032b8c
Boot0001-8be4df61-93ca-11d2-aa0d-00e098032b8c
Boot0003-8be4df61-93ca-11d2-aa0d-00e098032b8c
Boot0004-8be4df61-93ca-11d2-aa0d-00e098032b8c
Boot0005-8be4df61-93ca-11d2-aa0d-00e098032b8c
Boot0006-8be4df61-93ca-11d2-aa0d-00e098032b8c
BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
BootOptionSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c

I'm now trying to determine what changed in the kernel.