libdebian-installer uses a different detection method for EFI than efivar

[Benjamin Schreiber]

[Impact]
Some systems which do not correctly support EFI variables or have an incomplete implementation of the EFI spec may fail to install due to efibootmgr being unable to set BootEntry although archdetect reports the system being booted in EFI mode.

[Test case]
1) boot in EFI mode
TEST) Unmount /sys/firmware/efi/efivars and/or remove efivars module.
2) Run archdetect.

[Regression potential]
If efivars or vars don't get mounted (use of another init system than systemd, or lack of the efivars kernel module); the detection code will fail to see that the system is in EFI mode and installation will proceed in legacy mode.

--

After installing 16.04 (upgrade or re-install), I get this exception on login.

ProblemType: Package
DistroRelease: Ubuntu 16.04
Package: shim-signed 1.12+0.8-0ubuntu2
ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
ApportVersion: 2.20.1-0ubuntu2
Architecture: amd64
BootEFIContents:
 fw
 fwupx64.efi
Date: Mon May 16 09:28:29 2016
Df:
 Filesystem 1K-blocks Used Available Use% Mounted on
 /dev/sdb2 208735776 11061044 187048532 6% /
 /dev/sr0 1451056 1451056 0 100% /cdrom
 udev 7894540 0 7894540 0% /dev
 tmpfs 1581840 9536 1572304 1% /run
EFIBootMgr: Error: command ['efibootmgr', '-v'] failed with exit code 2: efibootmgr: EFI variables are not supported on this system.
ErrorMessage: subprocess installed post-installation script returned error exit status 1
RelatedPackageVersions:
 dpkg 1.18.4ubuntu1
 apt 1.2.10ubuntu1
SourcePackage: shim-signed
Title: package shim-signed 1.12+0.8-0ubuntu2 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1
UpgradeStatus: No upgrade log present (probably fresh install)

[Steve Langasek (vorlon)]

Sorry, when you say "upgrade or reinstall", what do you mean? You mean you saw this error on an upgrade, and also again after reinstalling the system?

The error message shows that you are installing the shim-signed package on a system that is not booted in EFI mode. We should fix this to not return an error (but such a fix is dependent on first fixing bug #1366546). But why is shim-signed being installed on your system, if the system is not booted in efi mode? Have you somehow requested this package?

The dpkg logs attached to your bug report are not particularly illuminating.

[Steve Langasek (vorlon)]

What I can see in the logs is this request for packages from the ubiquity installer:

Start-Date: 2016-05-16 09:28:12
Requested-By: ubuntu (999)
Install: intel-microcode:amd64 (3.20151106.1), iucode-tool:amd64 (1.5.1-1, automatic)
End-Date: 2016-05-16 09:28:19

Followed 5 seconds later by a request to install EFI grub:

Start-Date: 2016-05-16 09:28:24
Commandline: apt-get --no-upgrade -o Acquire::gpgv::Options::=--ignore-time-conflict -y install grub-efi-amd64-signed
Install: grub-efi-amd64-bin:amd64 (2.02~beta2-36ubuntu3, automatic), grub-efi-amd64:amd64 (2.02~beta2-36ubuntu3, automatic), grub-efi-amd64-signed:amd64 (1.66+2.02~beta2-36ubuntu3)
Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2016-05-16 09:28:26

It seems this must be an installer bug. Mathieu, can you please look at what is trying to install grub-efi on non-EFI systems and why?

[Mathieu Trudel-Lapierre (cyphermox)]

I'm not convinced that this isn't an EFI system.

The CD used here looks like it was booted in EFI mode:
[ 0.000000] Command line: BOOT_IMAGE=/casper/vmlinuz.efi file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity quiet splash ---

And that there was some EFI data available to the system as ESRT:
[ 0.000000] efi: EFI v2.31 by American Megatrends
[ 0.000000] efi: ESRT=0xbbf56898 ACPI=0xba7df000 ACPI 2.0=0xba7df000 SMBIOS=0xf04c0 MPS=0xfd4d0
[ 0.000000] esrt: Reserving ESRT space from 0x00000000bbf56898 to 0x00000000bbf568d0.

So we may be either running into a buggy implementation, or one of those 32-bit firmwares for a 64-bit system.

Could you please tell us what the make and model is for the computer on which you've received this error message? From this information we may be able to track down whether it's a known issue on that hardware.

[Benjamin Schreiber (v-ben-x)]

Thank you for looking into this. To clarify, I had attempted an in-place
update from 15.10 to 16.04. The update experienced some errors along the
way, but on reboot I was nonetheless in 16.04. But now I got a series of
error reports upon login (it never showed me the details).

In an attempt to resolve the problem, I attempted an OS reinstall, booting
from the LiveDVD for 16.04. In the process, I selected the option to
install 3rd-party drivers, and was prompted that I would need to disable
EFI (I have no idea whether EFI was previously active). I approved it,
then at the end of the installation was informed that Grub had failed to
install; the dialog that was supposed to then take me to a bug reporting
screen was then stuck, and couldn't be dismissed - I had to do a hard
reset. The startup after this is when I encountered the exception and used
the error reporting tool to open this ticket.

Unfortunately, I won't be able to provide you with any more details about
my system configuration, because I then re-ran the LiveDVD installer, this
time blowing away my system partition & installing clean (I had /home
mapped to a different physical volume, which proved very convenient in this
case). I did not select the option to install 3rd-party drivers, and my
system now boots and runs cleanly (I just have to re-install all my
optional software :( ).

On Mon, May 16, 2016 at 12:50 PM, Steve Langasek <
<email address hidden>> wrote:

> What I can see in the logs is this request for packages from the
> ubiquity installer:
>
> Start-Date: 2016-05-16 09:28:12
> Requested-By: ubuntu (999)
> Install: intel-microcode:amd64 (3.20151106.1), iucode-tool:amd64 (1.5.1-1,
> automatic)
> End-Date: 2016-05-16 09:28:19
>
> Followed 5 seconds later by a request to install EFI grub:
>
> Start-Date: 2016-05-16 09:28:24
> Commandline: apt-get --no-upgrade -o
> Acquire::gpgv::Options::=--ignore-time-conflict -y install
> grub-efi-amd64-signed
> Install: grub-efi-amd64-bin:amd64 (2.02~beta2-36ubuntu3, automatic),
> grub-efi-amd64:amd64 (2.02~beta2-36ubuntu3, automatic),
> grub-efi-amd64-signed:amd64 (1.66+2.02~beta2-36ubuntu3)
> Error: Sub-process /usr/bin/dpkg returned an error code (1)
> End-Date: 2016-05-16 09:28:26
>
> It seems this must be an installer bug. Mathieu, can you please look at
> what is trying to install grub-efi on non-EFI systems and why?
>
> ** Package changed: shim-signed (Ubuntu) => ubiquity (Ubuntu)
>
> ** Changed in: ubiquity (Ubuntu)
> Assignee: (unassigned) => Mathieu Trudel-Lapierre (cyphermox)
>
> ** Changed in: ubiquity (Ubuntu)
> Importance: Undecided => Critical
>
> ** Changed in: ubiquity (Ubuntu)
> Milestone: None => ubuntu-16.04.1
>
> ** Also affects: ubiquity (Ubuntu Yakkety)
> Importance: Critical
> Assignee: Mathieu Trudel-Lapierre (cyphermox)
> Status: Incomplete
>
> ** Also affects: ubiquity (Ubuntu Xenial)
> Importance: Undecided
> Status: New
>
> ** Changed in: ubiquity (Ubuntu Xenial)
> Importance: Undecided => Critical
>
> ** Changed in: ubiquity (Ubuntu Xenial)
> Assignee: (unassigned) => Mathieu Trudel-Lapierre (cyphermox)
>
> ** Changed in: ubiquity (Ubuntu Xenial)
...

[Benjamin Schreiber (v-ben-x)]

From taking a look at the motherboard manual, this is an EFI system (UEFI
BIOS). The system was built by a local independent integrator. The
motherboard is ASUS Q87M-E.

On Mon, May 16, 2016 at 1:15 PM, Mathieu Trudel-Lapierre <
<email address hidden>> wrote:

> I'm not convinced that this isn't an EFI system.
>
> The CD used here looks like it was booted in EFI mode:
> [ 0.000000] Command line: BOOT_IMAGE=/casper/vmlinuz.efi
> file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity quiet splash ---
>
> And that there was some EFI data available to the system as ESRT:
> [ 0.000000] efi: EFI v2.31 by American Megatrends
> [ 0.000000] efi: ESRT=0xbbf56898 ACPI=0xba7df000 ACPI
> 2.0=0xba7df000 SMBIOS=0xf04c0 MPS=0xfd4d0
> [ 0.000000] esrt: Reserving ESRT space from 0x00000000bbf56898 to
> 0x00000000bbf568d0.
>
> So we may be either running into a buggy implementation, or one of those
> 32-bit firmwares for a 64-bit system.
>
> Could you please tell us what the make and model is for the computer on
> which you've received this error message? From this information we may
> be able to track down whether it's a known issue on that hardware.
>
>
>
> ** Changed in: ubiquity (Ubuntu Xenial)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1582320
>
> Title:
> Problem detected upon boot/login
>
> Status in ubiquity package in Ubuntu:
> Incomplete
> Status in ubiquity source package in Xenial:
> Incomplete
> Status in ubiquity source package in Yakkety:
> Incomplete
>
> Bug description:
> After installing 16.04 (upgrade or re-install), I get this exception
> on login.
>
> ProblemType: Package
> DistroRelease: Ubuntu 16.04
> Package: shim-signed 1.12+0.8-0ubuntu2
> ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
> Uname: Linux 4.4.0-22-generic x86_64
> ApportVersion: 2.20.1-0ubuntu2
> Architecture: amd64
> BootEFIContents:
> fw
> fwupx64.efi
> Date: Mon May 16 09:28:29 2016
> Df:
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/sdb2 208735776 11061044 187048532 6% /
> /dev/sr0 1451056 1451056 0 100% /cdrom
> udev 7894540 0 7894540 0% /dev
> tmpfs 1581840 9536 1572304 1% /run
> EFIBootMgr: Error: command ['efibootmgr', '-v'] failed with exit code 2:
> efibootmgr: EFI variables are not supported on this system.
> ErrorMessage: subprocess installed post-installation script returned
> error exit status 1
> RelatedPackageVersions:
> dpkg 1.18.4ubuntu1
> apt 1.2.10ubuntu1
> SourcePackage: shim-signed
> Title: package shim-signed 1.12+0.8-0ubuntu2 failed to install/upgrade:
> subprocess installed post-installation script returned error exit status 1
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1582320/+subscriptions
>

[Mathieu Trudel-Lapierre (cyphermox)]

When you select to install 3rd-party drivers, you do not *have to* disable Secure Boot. We just recommend it because it is likely to be necessary for the 3rd-party drivers to work correctly.

So it sound like you've uncovered an issue in the installer for the third-party drivers case when we try to disable Secure Boot. I thought we were already avoiding any installer crashes there, but issues can and do happen, so thanks for reporting this bug!

There is just one bit of information we still don't have. Is this a custom-built computer or something bought from a known manufacturer? If it's a pre-made computer, could you share with us the manufacturer and model of the system?

[Steve Langasek (vorlon)]

On Mon, May 16, 2016 at 08:15:29PM -0000, Mathieu Trudel-Lapierre wrote:
> I'm not convinced that this isn't an EFI system.

> The CD used here looks like it was booted in EFI mode:
> [ 0.000000] Command line: BOOT_IMAGE=/casper/vmlinuz.efi file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity quiet splash ---

This is not evidence of an EFI boot. There is only a single vmlinuz image
on the CD, used by both BIOS and EFI boot. There is no reason to have two
separate vmlinuz images, and it doesn't matter that it's called
'vmlinuz.efi'.

> And that there was some EFI data available to the system as ESRT:
> [ 0.000000] efi: EFI v2.31 by American Megatrends
> [ 0.000000] efi: ESRT=0xbbf56898 ACPI=0xba7df000 ACPI 2.0=0xba7df000 SMBIOS=0xf04c0 MPS=0xfd4d0
> [ 0.000000] esrt: Reserving ESRT space from 0x00000000bbf56898 to 0x00000000bbf568d0.

I don't know how the kernel decides whether to initialize these drivers.
It's possible that this does point to the system having been booted under
EFI mode. But maybe it's also possible that the system was booted in CSM
mode.

[Mathieu Trudel-Lapierre (cyphermox)]

Oh, good point. I was operating under the false assumption that we did ship two separate copies of the kernel as on the installed system.

Still, there is no reason for the system to ever try to run efibootmgr / install grub-efi-amd64 unless it detects being booted in EFI mode, and where /sys/firmware/efi being available.

What we're getting here is that it may be that just checking if /sys/firmware/efi exists isn't sufficient to tell whether we're really on an EFI system.

Benjamin, if you boot the CD normally to get on the live system, could you then check if there are files under /sys/firmware/efi? If there are files populated by the kernel, but efivar reports that EFI variable support is absent (ie. some variable handling operations are unavailable), then we'll know what piece of the installer to look at.

[Phillip Susi (psusi)]

dmesg shows the bios e820 memory map and the dpkg history shows that grub-pc is currently installed, so yea, it is booting in bios mode and for some reason, is trying to switch to efi.

[Mathieu Trudel-Lapierre (cyphermox)]

Following your reinstallation, do you still get that error on boot/login?

What does running 'archdetect' report?

Also, could you please attach /var/log/installer/syslog on this bug report, in case we can get any information from it that would help in fixing the bug for others?

Thanks!

[Benjamin Schreiber (v-ben-x)]

As stated partway through the thread, after submitting the bug report through the automated tool, I wiped the boot partition & performed a clean install of 16.04, during which I did not select the option for 3rd party drivers (which I had selected previously). The resulting installation ran without incident, and I am up & running smoothly. For what it's worth, I've attached the file you requested from the most recent install, but unfortunately I neglected to capture any logs from the prior failed installations.

[Mathieu Trudel-Lapierre (cyphermox)]

Thanks.

Indeed, that install was clearly *not* in UEFI mode -- partman likely did not create an ESP, and the grub you have installed is grub-pc rather than grub-efi-amd64.

The efi and esrt kernel messages I had identified before and not there either.

Without reinstalling; would you mind trying to boot from DVD again as if you wanted to do another install? When you go boot from the DVD, are there multiple options in the boot menu to pick the DVD drive? For instance, one prefixed by "P0: " and one not, or some other difference? I would expect one entry to boot in UEFI, and thus show a grub menu rather than a prettier splash screen with the Ubuntu logo near the center; if you have the time and willingness to try and boot to "Try Ubuntu before installing" from the *grub* menu and running 'sudo efibootmgr -v' from a terminal there, that would be most helpful.

I'm kind of grasping at straws here, but we so far have not located a system on which we can reproduce the issue; the best bet to fix this for anybody else running into such a problem is to see whether there is something special about this BIOS.

[Mathieu Trudel-Lapierre (cyphermox)]

Even simpler than that, could you boot on a LiveCD/DVD (as above; trying to boot in UEFI mode); and tell us what are the contents of /sys/firmware/efi/efivars, and if it exists?

[Mathieu Trudel-Lapierre (cyphermox)]

Fix Released in yakkety:

libdebian-installer (0.102ubuntu3) yakkety; urgency=medium

  * src/system/efi.c: fix my blunder; we still need to have ret declared as an
    int since we need it for the ignore_uefi check. Derp.

libdebian-installer (0.102ubuntu2) yakkety; urgency=medium

  * src/system/efi.c: validate the presence of efivars *or* vars under
    /sys/firmware/efi to decide whether we should show the system as running
    in EFI mode; either of these paths is required for efibootmgr to set a
    BootEntry at the end of installation.

[Steve Langasek (vorlon)]

Hello Benjamin, or anyone else affected,

Accepted libdebian-installer into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libdebian-installer/0.102ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dimitri John Ledkov (xnox)]

debian-installer build that includes this udeb is ready to release for the s390x bug. Has this bug been verified? Is it ok to release d-i with this udeb built in into xenial-updates? Could you please verify this bug?

[Dimitri John Ledkov (xnox)]

All is good.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libdebian-installer - 0.102ubuntu1.1

---------------
libdebian-installer (0.102ubuntu1.1) xenial; urgency=medium

  * src/system/efi.c: validate the presence of efivars *or* vars under
    /sys/firmware/efi to decide whether we should show the system as running
    in EFI mode; either of these paths is required for efibootmgr to set a
    BootEntry at the end of installation. (LP: #1582320)

 -- Mathieu Trudel-Lapierre <email address hidden> Fri, 03 Jun 2016 21:24:31 -0400

[Martin Pitt (pitti)]

The verification of the Stable Release Update for libdebian-installer has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.