Secure Boot failure on Lenovo x3550 M5

[Rod Smith]

When doing certification testing of Ubuntu 16.04 on a Lenovo x3550 M5, we've found a Secure Boot failure. After installing via MAAS with Secure Boot DISABLED, we've enabled Secure Boot. The following appears on the screen (SOL session):

No key pressed. Preparing to boot normally...
>>Start PXE over IPv4.
  Station IP address is 10.1.10.17

  Server IP address is 10.1.10.1
  NBP filename is bootx64.efi
  NBP filesize is 1289424 Bytes
 Downloading NBP file...

  Succeed to download NBP file.

 Downloading NBP file...

  Succeed to download NBP file.
Fetching Netboot Image

Booting local disk...
/EndEntire
file path: /ACPI(a0341d0,0)/PCI(0,1)/PCI(0,0)/Ctrl(0)/SCSI(0,0)
/HD(15,800,100000,ae01bc523f0af546,2,2)/File(\efi\ubuntu)/File(shimx64.efi)/EndEntire
error: cannot load image.

Press any key to continue...

Pressing a key at this point produces a GRUB menu containing nothing but a "Local" option. Selecting that option causes a return of the "Booting local disk..." message and failure.

Disabling Secure Boot produces the same sequence, except that "error: cannot load image" does NOT appear, a GRUB menu with an "Ubuntu" option appears briefly, and the system boots normally.

Note that Secure Boot DOES work normally in a MAAS environment on other computers, such as Cisco C220 M4 and C240 M4 and an Intel NUC DC53427HYE. (The NUC, however, required a firmware update to work with Secure Boot active.)

This may well be a firmware bug, but I'm reporting it against Shim because it could be it's a Shim bug that's interacting with the firmware or there may be something Shim can do to work around the problem.

Version information:

$ lsb_release -rd
Description: Ubuntu 16.04 LTS
Release: 16.04
$ apt-cache policy shim
shim:
  Installed: 0.8-0ubuntu2
  Candidate: 0.8-0ubuntu2
  Version table:
 *** 0.8-0ubuntu2 500
        500 http://us.archive.ubuntu.com//ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status
ubuntu@oil-jolteon:~$ apt-cache policy shim-signed
shim-signed:
  Installed: 1.12+0.8-0ubuntu2
  Candidate: 1.12+0.8-0ubuntu2
  Version table:
 *** 1.12+0.8-0ubuntu2 500
        500 http://us.archive.ubuntu.com//ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status

[Steve Langasek (vorlon)]

Hi Rod,

On Thu, May 05, 2016 at 09:48:52PM -0000, Rod Smith wrote:
> When doing certification testing of Ubuntu 16.04 on a Lenovo x3550 M5,
> we've found a Secure Boot failure. After installing via MAAS with Secure
> Boot DISABLED, we've enabled Secure Boot. The following appears on the
> screen (SOL session):

> No key pressed. Preparing to boot normally...
> >>Start PXE over IPv4.
> Station IP address is 10.1.10.17
>
> Server IP address is 10.1.10.1
> NBP filename is bootx64.efi
> NBP filesize is 1289424 Bytes
> Downloading NBP file...

> Succeed to download NBP file.

Up to this point, this looks like a successful download of shim (named
bootx64.efi on the server) via PXE.

> Downloading NBP file...
>
> Succeed to download NBP file.

This looks like messages due to shim using the PXE protocol grabbing
grubx64.efi, and is also successful.

> Fetching Netboot Image

This is presumably output from grub, driven by whatever is in the grub.cfg.
Can you provide the grub.cfg that's present on the PXE server? Can you show
tftp logs of what files were downloaded, in what order?

> Booting local disk...
> /EndEntire
> file path: /ACPI(a0341d0,0)/PCI(0,1)/PCI(0,0)/Ctrl(0)/SCSI(0,0)
> /HD(15,800,100000,ae01bc523f0af546,2,2)/File(\efi\ubuntu)/File(shimx64.efi)/EndEntire
> error: cannot load image.

> Press any key to continue...

Are you expecting the system to boot from local disk at this point? Does
this path being booted from exist? Where does the path for this boot entry
come from, and if the file path exists, is it a proper copy of Ubuntu shim?

> Pressing a key at this point produces a GRUB menu containing nothing but
> a "Local" option. Selecting that option causes a return of the "Booting
> local disk..." message and failure.

I assume this is a boot menu from a grub.cfg provided by the MAAS server,
containing only a single boot entry pointing at the local disk and
configured to autoboot, so if the boot fails it takes you back to the same
menu again.

> Disabling Secure Boot produces the same sequence, except that "error:
> cannot load image" does NOT appear, a GRUB menu with an "Ubuntu" option
> appears briefly, and the system boots normally.

I assume this 'Ubuntu' option is the grub.cfg from the local disk displaying
briefly, only after the point that grub has successfully chainloaded to the
shim+grub on the local disk.

> Note that Secure Boot DOES work normally in a MAAS environment on other
> computers, such as Cisco C220 M4 and C240 M4 and an Intel NUC
> DC53427HYE. (The NUC, however, required a firmware update to work with
> Secure Boot active.)

> This may well be a firmware bug, but I'm reporting it against Shim
> because it could be it's a Shim bug that's interacting with the firmware
> or there may be something Shim can do to work around the problem.

Well, it could be a shim or grub bug, or a firmware bug, or even a maas bug,
depending on the above.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> ...

[Rod Smith (rodsmith)]

I'm attaching the contents of the /var/lib/maas/boot-resources/snapshot-20160426-183112 directory on the MAAS server (minus the ubuntu and custom directories, which contain OS images and are therefore huge). This tarball includes the shim and GRUB images used in this process. I'm also including an excerpt from the clusterd.log file from the MAAS server, which shows the TFTP requests.

You're correct that the system boots in two stages: When a node PXE-boots, it requests a boot loader from the MAAS server, which delivers an image that then kicks the boot process to the local disk. Thus, the boot process should be:

PXE request -> TFTP-delivered Shim -> TFTP-delivered GRUB -> local Shim -> local GRUB -> local kernel

At least, that's my understanding; I'm not involved in MAAS development in any significant way, so I could be misunderstanding something.

The symptoms look like the handoff from the TFTP-delivered GRUB to the local Shim is failing when Secure Boot is active. The file is definitely present because it DOES work when Secure Boot is inactive.

[Rod Smith (rodsmith)]

I've done some more experiments. After booting in the normal MAAS way with Secure Boot disabled, I created an entry with efibootmgr to boot directly from the local hard disk, bypassing MAAS and its PXE boot. The system booted fine with this configuration, even after enabling Secure Boot. Thus, this is not a blanket Secure Boot issue; it has something to do with the handoff through multiple programs used by the MAAS configuration. (I expected this to be the case because the first GRUB delivered by MAAS booted OK; however, I know of at least one computer that ignores Secure Boot for network boots.)

I then modified /boot/grub/grub.cfg to add entries to chainload from the disk-based GRUB to itself, both directly and via the on-disk shimx64.efi. Both these entries failed, with the same message noted earlier:

/EndEntire
file path: /ACPI(a0341d0,0)/PCI(0,1)/PCI(0,0)/Ctrl(0)/SCSI(0,0)
/HD(15,800,100000,ae01bc523f0af546,2,2)/File(\efi\ubuntu)/File(shimx64.efi)/EndEntire
error: cannot load image.

(In the case of booting GRUB directly, of course, the message referenced grubx64.efi, not shimx64.efi.)

I then installed rEFInd on the system. This starts fine, and can redirect to grubx64.efi, which also starts fine and can launch a kernel; however, if rEFInd boots shimx64.efi, although a GRUB menu appears, GRUB refuses to load the kernel, with the following message:

Bootloader has not verified loaded image.
System is compromised. halting.

rEFInd can also directly launch a signed Linux kernel, but not an unsigned one. This is normal and expected behavior for rEFInd.

Thus, this appears to be a problem with shim verifying a second or subsequent image and/or with something in the way GRUB is launching chainloaded EFI programs. I ran into a similar problem with shim 0.8 and rEFInd, which I worked around with an ugly hack in rEFInd 0.9.2; however, I'm also reminded of bug 1091464, which has existed for much longer and that may be related to this problem. In that bug, GRUB is unable to chainload to Windows when Secure Boot is active. As the broken point here seems to be GRUB launching shim, the problem seems analogous.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.