UEFI in ovmf package causes kernel panic

Bug #1821729 reported by Riccardo Pittau on 2019-03-26

This bug report will be marked for expiration in 48 days if no further activity occurs. (find out why)

8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
edk2 (Ubuntu)
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned

Bug Description

UBUNTU info
Description: Ubuntu 18.04.1 LTS
Release: 18.04

PACKAGE info
ovmf:
  Installed: 0~20180205.c0d9813c-2
  Candidate: 0~20180205.c0d9813c-2
  Version table:
 *** 0~20180205.c0d9813c-2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
        100 /var/lib/dpkg/status

Expected:
Virtual machines loading from the UEFI provided by the ovmf package should work fine.
This is working correctly in Ubuntu Xenial with ovmf package 0~20160408.ffea0a2c-2 that provides EFI v2.60 by EDK II.

What happens:
Virtual machines loading from the UEFI crash with kernel panic.
Provided UEFI version: EFI v2.70 by EDK II.

To be able to correctly boot up a VM on bionic, we downgraded the package.
Here's an example from a succesfull automated devstack job:
https://review.openstack.org/647687 -> ironic-tempest-ipa-partition-uefi-pxe_ipmitool-tinyipa

dann frazier (dannf) wrote :

Please provide the console log and libvirt xml for a failing VM.

Changed in edk2 (Ubuntu):
status: New → Incomplete
Riccardo Pittau (rpittau) wrote :

console log attached

Riccardo Pittau (rpittau) wrote :

libvirt xml of virtual node attached

dann frazier (dannf) wrote :

@Riccardo: Thanks, I can reproduce by using machine='pc-1.0' in my XML (pc-i440fx-2.12 worked fine). I also found that the version of ovmf in disco works w/ pc-1.0, so I'll try to bisect down what fixed it.

dann frazier (dannf) wrote :

Bisection led me to the following fix:
 https://github.com/tianocore/edk2/commit/272658b9971865812f70e494d0198f13df8b841b

I've uploaded test kernels to a PPA, can you verify that the 18.04 ('bionic') one fixes the problem for you?
  https://launchpad.net/~dannf/+archive/ubuntu/test

Changed in edk2 (Ubuntu Cosmic):
status: New → Incomplete
Changed in edk2 (Ubuntu Bionic):
status: New → Incomplete
Riccardo Pittau (rpittau) wrote :

@Dann: thanks for checking that, I did some tests and, even if I can't reproduce the kernel panic anymore, I'm seeing another issue that was also happening before with virtio, the vm is not booting on ipxe.

This is the correct output on bionic using virtio + ovmf from xenial:
http://logs.openstack.org/87/647687/5/check/ironic-tempest-ipa-partition-uefi-pxe_ipmitool-tinyipa/a16650a/controller/logs/ironic-bm-logs/node-0_no_ansi_2019-03-27-14:18:35_log.txt.gz

This is a test done with virtio and the new package on bionic:
http://logs.openstack.org/07/645507/7/check/ironic-tempest-ipa-partition-uefi-pxe_ipmitool-tinyipa/2660087/controller/logs/ironic-bm-logs/node-0_no_ansi_2019-04-01-09:57:23_log.txt.gz

This is the output changing the driver to e1000 and using the default rom on bionic:
http://logs.openstack.org/38/648938/1/check/ironic-tempest-ipa-partition-uefi-pxe_ipmitool-tinyipa/c943520/controller/logs/ironic-bm-logs/node-0_no_ansi_2019-04-01-11:01:55_log.txt.gz

This is the output using e1000 driver using the rom from ipxe-qemu on bionic:
http://logs.openstack.org/38/648938/2/check/ironic-tempest-ipa-partition-uefi-pxe_ipmitool-tinyipa/1c53407/controller/logs/ironic-bm-logs/node-0_no_ansi_2019-04-01-13:15:39_log.txt.gz

I reproduced all the tests locally to confirm, also changing ipxe-qemu from bionic to xenial to understand if it was a regression in that package, but it didn't work, so I think this is another problem in recent versions of ovmf.

dann frazier (dannf) wrote :

@Riccardo: Sorry, I'm not sure how to read your update. Are you stating that you are unable to reproduce the panic *after installing a PPA build*, or for some other reason?

Can you confirm that the ipxe thing is a separate issue? If so, please file a new bug for that instead of overloading this one.

Riccardo Pittau (rpittau) wrote :

@Dann: what I'm saying is that I can't reproduce the kernel panic after installing a PPA build because it doesn't load from ipxe anymore.

I can't confirm or deny the issue with the kernel panic is actually fixed.

If you prefer I can open another bug, although it seems a regression to me, but of course a different issue.

On Mon, Apr 1, 2019 at 9:50 AM Riccardo Pittau
<email address hidden> wrote:
>
> @Dann: what I'm saying is that I can't reproduce the kernel panic after
> installing a PPA build because it doesn't load from ipxe anymore.
>
> I can't confirm or deny the issue with the kernel panic is actually
> fixed.
>
> If you prefer I can open another bug, although it seems a regression to
> me, but of course a different issue.

OK, I understand now - thanks for clarifying.

  -dann

dann frazier (dannf) wrote :

@Riccardo: The PPA builds I provided had both a fix for this issue and some security issues, some related to networking code. To help determine if it is the security fixes or the fix for this bug causing your iPXE regression, I've uploaded new packages *without* the security fix. Could you test those? They are in the same PPA.

Unfortunately, I'm unable to reproduce the iPXE issue myself. With ovmf 0~20180205.c0d9813c-2ubuntu0.1~ppa.1 from my PPA, I'm able to boot into iPXE just fine:

https://paste.ubuntu.com/p/T5Pp7GCmrH/

I haven't configured iPXE to actually boot a kernel, so it stops there - but yours appears to hang well before that. I've actually only used iPXE a few times - what I'm doing is using the internal virtio PXE support from ovmf to PXE boot the iPXE payload. I don't see any messages in your log prior to iPXE - are you doing the same thing?

Riccardo Pittau (rpittau) wrote :

@Dann: thanks for keep looking at this, I tested the new package both locally and in our CI, using virtio and e1000 drivers, but unfortunately I'm still seeing the issue where the vm can't load from ipxe.
The only thing that we change for the test is the ovmf package, the rest of the configuration is exactly the same.
I'm quite puzzled at this point seeing that you were actually able to correctly load from ipxe, in my case it seems the vm hangs before getting an ip during the pxeboot phase.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers