| 2020-06-09 10:42:45 |
Vladislav K. Valtchev |
bug |
|
|
added bug |
| 2020-06-09 10:43:54 |
Vladislav K. Valtchev |
bug task added |
|
qemu (Ubuntu) |
|
| 2020-06-09 11:12:30 |
Philippe Mathieu-Daudé |
bug |
|
|
added subscriber Gerd Hoffmann |
| 2020-06-09 19:06:34 |
Vladislav K. Valtchev |
attachment added |
|
QEMU's debug log file https://bugs.launchpad.net/qemu/+bug/1882671/+attachment/5382085/+files/debug.log |
|
| 2020-06-10 10:18:13 |
Launchpad Janitor |
qemu (Ubuntu): status |
New |
Confirmed |
|
| 2020-06-10 23:18:21 |
Laszlo Ersek (Red Hat) |
affects |
qemu |
ipxe |
|
| 2020-06-10 23:18:44 |
Laszlo Ersek (Red Hat) |
affects |
qemu (Ubuntu) |
ipxe (Ubuntu) |
|
| 2020-06-10 23:35:13 |
Laszlo Ersek (Red Hat) |
summary |
qemu-system-x86_64 (ver 4.2) stuck at boot with OVMF bios |
unbalanced UEFI TPL manipulations in iPXE with DOWNLOAD_PROTO_HTTPS enabled |
|
| 2020-06-11 07:22:02 |
Philippe Mathieu-Daudé |
removed subscriber Gerd Hoffmann |
|
|
|
| 2020-06-15 07:01:32 |
Christian Ehrhardt |
bug |
|
|
added subscriber Christian Ehrhardt |
| 2020-06-16 18:41:12 |
Sergio Durigan Junior |
bug |
|
|
added subscriber Ubuntu Server |
| 2020-06-19 17:44:59 |
Laszlo Ersek (Red Hat) |
bug |
|
|
added subscriber Laszlo Ersek (Red Hat) |
| 2020-06-24 10:09:29 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Focal |
|
| 2020-06-24 10:09:29 |
Christian Ehrhardt |
bug task added |
|
ipxe (Ubuntu Focal) |
|
| 2020-06-24 10:09:29 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Eoan |
|
| 2020-06-24 10:09:29 |
Christian Ehrhardt |
bug task added |
|
ipxe (Ubuntu Eoan) |
|
| 2020-06-25 07:21:34 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/ipxe/+git/ipxe/+merge/386372 |
|
| 2020-06-25 11:42:59 |
Christian Ehrhardt |
attachment added |
|
OVMF build with debug to verify the bug https://bugs.launchpad.net/ubuntu/+source/ipxe/+bug/1882671/+attachment/5387066/+files/OVMF-14c7ed8b51f6-DEBUG-enabled.fd |
|
| 2020-06-25 11:43:07 |
Christian Ehrhardt |
ipxe (Ubuntu): status |
Confirmed |
In Progress |
|
| 2020-06-25 11:43:09 |
Christian Ehrhardt |
ipxe (Ubuntu Focal): status |
New |
Triaged |
|
| 2020-06-25 11:43:11 |
Christian Ehrhardt |
ipxe (Ubuntu Eoan): status |
New |
Triaged |
|
| 2020-06-25 11:43:14 |
Christian Ehrhardt |
ipxe (Ubuntu Eoan): importance |
Undecided |
Low |
|
| 2020-06-25 11:43:16 |
Christian Ehrhardt |
ipxe (Ubuntu Focal): importance |
Undecided |
Low |
|
| 2020-06-25 11:43:18 |
Christian Ehrhardt |
ipxe (Ubuntu): importance |
Undecided |
High |
|
| 2020-06-25 13:55:11 |
Christian Ehrhardt |
description |
The version of QEMU (4.2.0) packaged for Ubuntu 20.04 hangs indefinitely at boot if an OVMF bios is used. This happens ONLY with qemu-system-x86_64. qemu-system-i386 works fine with the latest ia32 OVMF bios.
NOTE[1]: the same identical OVMF bios works fine on QEMU 2.x packaged with Ubuntu 18.04.
NOTE[2]: reproducing the fatal bug requires *no* operating system:
qemu-system-x86_64 -bios OVMF-pure-efi.fd
On its window QEMU gets stuck at the very first stage:
"Guest has not initialized the display (yet)."
NOTE[3]: QEMU gets stuck no matter if KVM is used or not.
NOTE[4]: By adding the `-d int` option it is possible to observe that QEMU is, apparently, stuck in an endless loop of interrupts. For the first few seconds, registers' values vary quickly, but at some point they reach a final value, while the interrupt counter increments:
2568: v=68 e=0000 i=0 cpl=0 IP=0038:0000000007f1d225 pc=0000000007f1d225 SP=0030:0000000007f0c8d0 env->regs[R_EAX]=0000000000000000
RAX=0000000000000000 RBX=0000000007f0c920 RCX=0000000000000000 RDX=0000000000000001
RSI=0000000006d18798 RDI=0000000000008664 RBP=0000000000000000 RSP=0000000007f0c8d0
R8 =0000000000000001 R9 =0000000000000089 R10=0000000000000000 R11=0000000007f2c987
R12=0000000000000000 R13=0000000000000000 R14=0000000007087901 R15=0000000000000000
RIP=0000000007f1d225 RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
CS =0038 0000000000000000 ffffffff 00af9a00 DPL=0 CS64 [-R-]
SS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
DS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
FS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
GS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 00000000079eea98 00000047
IDT= 000000000758f018 00000fff
CR0=80010033 CR2=0000000000000000 CR3=0000000007c01000 CR4=00000668
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000044 CCD=0000000000000000 CCO=EFLAGS
EFER=0000000000000d00
NOTE[5]: Just to better help the investigation of the bug, I'd like to remark that the issue is NOT caused by an endless loop of triple-faults. I tried with -d cpu_reset and there is NO such loop. No triple fault whatsoever.
NOTE[6]: The OVMF version used for the test has been downloaded from:
https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20200515.1398.g6ff7c838d0.noarch.rpm
but the issue is the same with older OVMF versions as well.
Please take a look at it, as the bug is NOT a corner case. QEMU 4.2.0 cannot boot with an UEFI firmware (OVMF) while virtualizing a x86_64 machine AT ALL.
Thank you very much,
Vladislav K. Valtchev |
[Impact]
* Booting some OVMF through the efi roms in our ipxe-qemu package triggers
a bad ordering of TPL manipulations and eventually gets the boot stuck.
* It was analyzed by Lazlo that this was due to HTTPS being enabled in the
rom itself triggering this fail when gathering entropy. In addition
isn't required to have HTTPS enabled in this part of the roms as the EFI
payload would be able to handle that on it's own (inside OVMF file from
src:edk2 instead of the combined rom like efi-e1000.rom).
* Disabling HTTPS in the efi-*.rom files should have no functional
impact but mitigate the TPL issues we are seeing.
[Test Case]
* I have provided a test OVMF load attached to this bug
https://bugs.launchpad.net/ipxe/+bug/1882671/+attachment/5387066/+files/OVMF-14c7ed8b51f6-DEBUG-enabled.fd
* Start that in qemu like:
$ qemu-system-x86_64 -enable-kvm -monitor stdio -drive if=pflash,snapshot=on,format=raw,file=/root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd -global e1000.romfile=/usr/lib/ipxe/qemu/efi-e1000.rom -debugcon file:debug.log -global isa-debugcon.iobase=0x402
# /usr/lib/ipxe/qemu/efi-e1000.rom is from package ipxe-qemu
# /root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd is the debug build I
attached
Symptoms when failing:
- UI never leaves the "initializing" state
- in the debug.log the bad case asserts with
ASSERT /root/edk2/MdeModulePkg/Core/Dxe/Image/Image.c(1676):
Image->Tpl == gEfiCurrentTpl
* Extra Test: HTTPS boot a uEFI guest with the efi roms from ipxe-qemu
with old/new code
[Regression Potential]
* Messing around with rom options always is a complex topic, but I'd like
to quote Lazlo (who understands all of this better anyway):
"you don't need the iPXE HTTPS implementation, because OVMF already contains the edk2 HTTPS BOOT feature (assuming you build OVMF with "-D NETWORK_HTTP_BOOT_ENABLE -D NETWORK_TLS_ENABLE"). The whole point of CONFIG=qemu is to strip the UEFI build of iPXE to a mere Simple Network Protocol driver. Such a build provides only the lowest level hardware abstraction for UEFI, and the rest (IPv[46], TCP/UDP, DHCP, PXE, TFTP, HTTP(S)) are provided on top by the edk2 implementations of various standard UEFI interfaces."
* Never the less, if you seek regressions then in the UEFI handling of
HTTPS boot.
[Other Info]
* This happens with some OVMF versions like the one I have provided for
debugging but not with the OVMF delivered as part of Focal. Therefore we
should fix it but priority isn't that high.
* Quote from the discussion: "If you want to run a full-featured iPXE
build on a UEFI machine (including: in an OVMF guest), you still can,
of course; lots of people do that, for good reasons. But that use case
is best served by the *standalone UEFI application* build of iPXE
(produced at "bin-x86_64-efi/ipxe.efi" by "make"). The UEFI *driver*
build of iPXE should be as minimal as possible, in comparison -- just
provide SNP for the desired NIC models."
=> Only the EFI roms are changed, neither the PCI roms nor the standalon
IPXE builds will be modified.
---
The version of QEMU (4.2.0) packaged for Ubuntu 20.04 hangs indefinitely at boot if an OVMF bios is used. This happens ONLY with qemu-system-x86_64. qemu-system-i386 works fine with the latest ia32 OVMF bios.
NOTE[1]: the same identical OVMF bios works fine on QEMU 2.x packaged with Ubuntu 18.04.
NOTE[2]: reproducing the fatal bug requires *no* operating system:
qemu-system-x86_64 -bios OVMF-pure-efi.fd
On its window QEMU gets stuck at the very first stage:
"Guest has not initialized the display (yet)."
NOTE[3]: QEMU gets stuck no matter if KVM is used or not.
NOTE[4]: By adding the `-d int` option it is possible to observe that QEMU is, apparently, stuck in an endless loop of interrupts. For the first few seconds, registers' values vary quickly, but at some point they reach a final value, while the interrupt counter increments:
2568: v=68 e=0000 i=0 cpl=0 IP=0038:0000000007f1d225 pc=0000000007f1d225 SP=0030:0000000007f0c8d0 env->regs[R_EAX]=0000000000000000
RAX=0000000000000000 RBX=0000000007f0c920 RCX=0000000000000000 RDX=0000000000000001
RSI=0000000006d18798 RDI=0000000000008664 RBP=0000000000000000 RSP=0000000007f0c8d0
R8 =0000000000000001 R9 =0000000000000089 R10=0000000000000000 R11=0000000007f2c987
R12=0000000000000000 R13=0000000000000000 R14=0000000007087901 R15=0000000000000000
RIP=0000000007f1d225 RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
CS =0038 0000000000000000 ffffffff 00af9a00 DPL=0 CS64 [-R-]
SS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
DS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
FS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
GS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 00000000079eea98 00000047
IDT= 000000000758f018 00000fff
CR0=80010033 CR2=0000000000000000 CR3=0000000007c01000 CR4=00000668
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000044 CCD=0000000000000000 CCO=EFLAGS
EFER=0000000000000d00
NOTE[5]: Just to better help the investigation of the bug, I'd like to remark that the issue is NOT caused by an endless loop of triple-faults. I tried with -d cpu_reset and there is NO such loop. No triple fault whatsoever.
NOTE[6]: The OVMF version used for the test has been downloaded from:
https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20200515.1398.g6ff7c838d0.noarch.rpm
but the issue is the same with older OVMF versions as well.
Please take a look at it, as the bug is NOT a corner case. QEMU 4.2.0 cannot boot with an UEFI firmware (OVMF) while virtualizing a x86_64 machine AT ALL.
Thank you very much,
Vladislav K. Valtchev |
|
| 2020-06-30 18:16:06 |
Launchpad Janitor |
ipxe (Ubuntu): status |
In Progress |
Fix Released |
|
| 2020-07-01 06:06:25 |
Christian Ehrhardt |
description |
[Impact]
* Booting some OVMF through the efi roms in our ipxe-qemu package triggers
a bad ordering of TPL manipulations and eventually gets the boot stuck.
* It was analyzed by Lazlo that this was due to HTTPS being enabled in the
rom itself triggering this fail when gathering entropy. In addition
isn't required to have HTTPS enabled in this part of the roms as the EFI
payload would be able to handle that on it's own (inside OVMF file from
src:edk2 instead of the combined rom like efi-e1000.rom).
* Disabling HTTPS in the efi-*.rom files should have no functional
impact but mitigate the TPL issues we are seeing.
[Test Case]
* I have provided a test OVMF load attached to this bug
https://bugs.launchpad.net/ipxe/+bug/1882671/+attachment/5387066/+files/OVMF-14c7ed8b51f6-DEBUG-enabled.fd
* Start that in qemu like:
$ qemu-system-x86_64 -enable-kvm -monitor stdio -drive if=pflash,snapshot=on,format=raw,file=/root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd -global e1000.romfile=/usr/lib/ipxe/qemu/efi-e1000.rom -debugcon file:debug.log -global isa-debugcon.iobase=0x402
# /usr/lib/ipxe/qemu/efi-e1000.rom is from package ipxe-qemu
# /root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd is the debug build I
attached
Symptoms when failing:
- UI never leaves the "initializing" state
- in the debug.log the bad case asserts with
ASSERT /root/edk2/MdeModulePkg/Core/Dxe/Image/Image.c(1676):
Image->Tpl == gEfiCurrentTpl
* Extra Test: HTTPS boot a uEFI guest with the efi roms from ipxe-qemu
with old/new code
[Regression Potential]
* Messing around with rom options always is a complex topic, but I'd like
to quote Lazlo (who understands all of this better anyway):
"you don't need the iPXE HTTPS implementation, because OVMF already contains the edk2 HTTPS BOOT feature (assuming you build OVMF with "-D NETWORK_HTTP_BOOT_ENABLE -D NETWORK_TLS_ENABLE"). The whole point of CONFIG=qemu is to strip the UEFI build of iPXE to a mere Simple Network Protocol driver. Such a build provides only the lowest level hardware abstraction for UEFI, and the rest (IPv[46], TCP/UDP, DHCP, PXE, TFTP, HTTP(S)) are provided on top by the edk2 implementations of various standard UEFI interfaces."
* Never the less, if you seek regressions then in the UEFI handling of
HTTPS boot.
[Other Info]
* This happens with some OVMF versions like the one I have provided for
debugging but not with the OVMF delivered as part of Focal. Therefore we
should fix it but priority isn't that high.
* Quote from the discussion: "If you want to run a full-featured iPXE
build on a UEFI machine (including: in an OVMF guest), you still can,
of course; lots of people do that, for good reasons. But that use case
is best served by the *standalone UEFI application* build of iPXE
(produced at "bin-x86_64-efi/ipxe.efi" by "make"). The UEFI *driver*
build of iPXE should be as minimal as possible, in comparison -- just
provide SNP for the desired NIC models."
=> Only the EFI roms are changed, neither the PCI roms nor the standalon
IPXE builds will be modified.
---
The version of QEMU (4.2.0) packaged for Ubuntu 20.04 hangs indefinitely at boot if an OVMF bios is used. This happens ONLY with qemu-system-x86_64. qemu-system-i386 works fine with the latest ia32 OVMF bios.
NOTE[1]: the same identical OVMF bios works fine on QEMU 2.x packaged with Ubuntu 18.04.
NOTE[2]: reproducing the fatal bug requires *no* operating system:
qemu-system-x86_64 -bios OVMF-pure-efi.fd
On its window QEMU gets stuck at the very first stage:
"Guest has not initialized the display (yet)."
NOTE[3]: QEMU gets stuck no matter if KVM is used or not.
NOTE[4]: By adding the `-d int` option it is possible to observe that QEMU is, apparently, stuck in an endless loop of interrupts. For the first few seconds, registers' values vary quickly, but at some point they reach a final value, while the interrupt counter increments:
2568: v=68 e=0000 i=0 cpl=0 IP=0038:0000000007f1d225 pc=0000000007f1d225 SP=0030:0000000007f0c8d0 env->regs[R_EAX]=0000000000000000
RAX=0000000000000000 RBX=0000000007f0c920 RCX=0000000000000000 RDX=0000000000000001
RSI=0000000006d18798 RDI=0000000000008664 RBP=0000000000000000 RSP=0000000007f0c8d0
R8 =0000000000000001 R9 =0000000000000089 R10=0000000000000000 R11=0000000007f2c987
R12=0000000000000000 R13=0000000000000000 R14=0000000007087901 R15=0000000000000000
RIP=0000000007f1d225 RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
CS =0038 0000000000000000 ffffffff 00af9a00 DPL=0 CS64 [-R-]
SS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
DS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
FS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
GS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 00000000079eea98 00000047
IDT= 000000000758f018 00000fff
CR0=80010033 CR2=0000000000000000 CR3=0000000007c01000 CR4=00000668
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000044 CCD=0000000000000000 CCO=EFLAGS
EFER=0000000000000d00
NOTE[5]: Just to better help the investigation of the bug, I'd like to remark that the issue is NOT caused by an endless loop of triple-faults. I tried with -d cpu_reset and there is NO such loop. No triple fault whatsoever.
NOTE[6]: The OVMF version used for the test has been downloaded from:
https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20200515.1398.g6ff7c838d0.noarch.rpm
but the issue is the same with older OVMF versions as well.
Please take a look at it, as the bug is NOT a corner case. QEMU 4.2.0 cannot boot with an UEFI firmware (OVMF) while virtualizing a x86_64 machine AT ALL.
Thank you very much,
Vladislav K. Valtchev |
[Impact]
* Booting some OVMF through the efi roms in our ipxe-qemu package triggers
a bad ordering of TPL manipulations and eventually gets the boot stuck.
* It was analyzed by Lazlo that this was due to HTTPS being enabled in the
rom itself triggering this fail when gathering entropy. In addition
isn't required to have HTTPS enabled in this part of the roms as the EFI
payload would be able to handle that on it's own (inside OVMF file from
src:edk2 instead of the combined rom like efi-e1000.rom).
* Disabling HTTPS in the efi-*.rom files should have no functional
impact but mitigate the TPL issues we are seeing.
[Test Case]
* I have provided a test OVMF load attached to this bug
https://bugs.launchpad.net/ipxe/+bug/1882671/+attachment/5387066/+files/OVMF-14c7ed8b51f6-DEBUG-enabled.fd
* Start that in qemu like:
$ qemu-system-x86_64 -enable-kvm -monitor stdio -drive if=pflash,snapshot=on,format=raw,file=/root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd -global e1000.romfile=/usr/lib/ipxe/qemu/efi-e1000.rom -debugcon file:debug.log -global isa-debugcon.iobase=0x402
# /usr/lib/ipxe/qemu/efi-e1000.rom is from package ipxe-qemu
# /root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd is the debug build I
attached
Symptoms when failing:
- UI never leaves the "initializing" state
- in the debug.log the bad case asserts with
ASSERT /root/edk2/MdeModulePkg/Core/Dxe/Image/Image.c(1676):
Image->Tpl == gEfiCurrentTpl
* Extra Test: HTTPS boot a uEFI guest with the efi roms from ipxe-qemu
with old/new ipxe-qmeu code. This shall ensure that OVMF can really take
over as-is or if we need bug 1883114 before we can do so.
Details TBD
[Regression Potential]
* Messing around with rom options always is a complex topic, but I'd like
to quote Lazlo (who understands all of this better anyway):
"you don't need the iPXE HTTPS implementation, because OVMF already contains the edk2 HTTPS BOOT feature (assuming you build OVMF with "-D NETWORK_HTTP_BOOT_ENABLE -D NETWORK_TLS_ENABLE"). The whole point of CONFIG=qemu is to strip the UEFI build of iPXE to a mere Simple Network Protocol driver. Such a build provides only the lowest level hardware abstraction for UEFI, and the rest (IPv[46], TCP/UDP, DHCP, PXE, TFTP, HTTP(S)) are provided on top by the edk2 implementations of various standard UEFI interfaces."
* Never the less, if you seek regressions then in the UEFI handling of
HTTPS boot.
[Other Info]
* This happens with some OVMF versions like the one I have provided for
debugging but not with the OVMF delivered as part of Focal. Therefore we
should fix it but priority isn't that high.
* Quote from the discussion: "If you want to run a full-featured iPXE
build on a UEFI machine (including: in an OVMF guest), you still can,
of course; lots of people do that, for good reasons. But that use case
is best served by the *standalone UEFI application* build of iPXE
(produced at "bin-x86_64-efi/ipxe.efi" by "make"). The UEFI *driver*
build of iPXE should be as minimal as possible, in comparison -- just
provide SNP for the desired NIC models."
=> Only the EFI roms are changed, neither the PCI roms nor the standalon
IPXE builds will be modified.
---
The version of QEMU (4.2.0) packaged for Ubuntu 20.04 hangs indefinitely at boot if an OVMF bios is used. This happens ONLY with qemu-system-x86_64. qemu-system-i386 works fine with the latest ia32 OVMF bios.
NOTE[1]: the same identical OVMF bios works fine on QEMU 2.x packaged with Ubuntu 18.04.
NOTE[2]: reproducing the fatal bug requires *no* operating system:
qemu-system-x86_64 -bios OVMF-pure-efi.fd
On its window QEMU gets stuck at the very first stage:
"Guest has not initialized the display (yet)."
NOTE[3]: QEMU gets stuck no matter if KVM is used or not.
NOTE[4]: By adding the `-d int` option it is possible to observe that QEMU is, apparently, stuck in an endless loop of interrupts. For the first few seconds, registers' values vary quickly, but at some point they reach a final value, while the interrupt counter increments:
2568: v=68 e=0000 i=0 cpl=0 IP=0038:0000000007f1d225 pc=0000000007f1d225 SP=0030:0000000007f0c8d0 env->regs[R_EAX]=0000000000000000
RAX=0000000000000000 RBX=0000000007f0c920 RCX=0000000000000000 RDX=0000000000000001
RSI=0000000006d18798 RDI=0000000000008664 RBP=0000000000000000 RSP=0000000007f0c8d0
R8 =0000000000000001 R9 =0000000000000089 R10=0000000000000000 R11=0000000007f2c987
R12=0000000000000000 R13=0000000000000000 R14=0000000007087901 R15=0000000000000000
RIP=0000000007f1d225 RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
CS =0038 0000000000000000 ffffffff 00af9a00 DPL=0 CS64 [-R-]
SS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
DS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
FS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
GS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 00000000079eea98 00000047
IDT= 000000000758f018 00000fff
CR0=80010033 CR2=0000000000000000 CR3=0000000007c01000 CR4=00000668
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000044 CCD=0000000000000000 CCO=EFLAGS
EFER=0000000000000d00
NOTE[5]: Just to better help the investigation of the bug, I'd like to remark that the issue is NOT caused by an endless loop of triple-faults. I tried with -d cpu_reset and there is NO such loop. No triple fault whatsoever.
NOTE[6]: The OVMF version used for the test has been downloaded from:
https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20200515.1398.g6ff7c838d0.noarch.rpm
but the issue is the same with older OVMF versions as well.
Please take a look at it, as the bug is NOT a corner case. QEMU 4.2.0 cannot boot with an UEFI firmware (OVMF) while virtualizing a x86_64 machine AT ALL.
Thank you very much,
Vladislav K. Valtchev |
|
| 2020-07-01 06:15:54 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/ipxe/+git/ipxe/+merge/386647 |
|
| 2020-07-01 06:17:59 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/ipxe/+git/ipxe/+merge/386648 |
|
| 2020-07-03 05:15:13 |
Christian Ehrhardt |
ipxe (Ubuntu Eoan): status |
Triaged |
Won't Fix |
|
| 2020-07-03 05:17:53 |
Christian Ehrhardt |
tags |
|
block-proposed block-proposed-focal |
|
| 2020-07-03 05:20:30 |
Christian Ehrhardt |
description |
[Impact]
* Booting some OVMF through the efi roms in our ipxe-qemu package triggers
a bad ordering of TPL manipulations and eventually gets the boot stuck.
* It was analyzed by Lazlo that this was due to HTTPS being enabled in the
rom itself triggering this fail when gathering entropy. In addition
isn't required to have HTTPS enabled in this part of the roms as the EFI
payload would be able to handle that on it's own (inside OVMF file from
src:edk2 instead of the combined rom like efi-e1000.rom).
* Disabling HTTPS in the efi-*.rom files should have no functional
impact but mitigate the TPL issues we are seeing.
[Test Case]
* I have provided a test OVMF load attached to this bug
https://bugs.launchpad.net/ipxe/+bug/1882671/+attachment/5387066/+files/OVMF-14c7ed8b51f6-DEBUG-enabled.fd
* Start that in qemu like:
$ qemu-system-x86_64 -enable-kvm -monitor stdio -drive if=pflash,snapshot=on,format=raw,file=/root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd -global e1000.romfile=/usr/lib/ipxe/qemu/efi-e1000.rom -debugcon file:debug.log -global isa-debugcon.iobase=0x402
# /usr/lib/ipxe/qemu/efi-e1000.rom is from package ipxe-qemu
# /root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd is the debug build I
attached
Symptoms when failing:
- UI never leaves the "initializing" state
- in the debug.log the bad case asserts with
ASSERT /root/edk2/MdeModulePkg/Core/Dxe/Image/Image.c(1676):
Image->Tpl == gEfiCurrentTpl
* Extra Test: HTTPS boot a uEFI guest with the efi roms from ipxe-qemu
with old/new ipxe-qmeu code. This shall ensure that OVMF can really take
over as-is or if we need bug 1883114 before we can do so.
Details TBD
[Regression Potential]
* Messing around with rom options always is a complex topic, but I'd like
to quote Lazlo (who understands all of this better anyway):
"you don't need the iPXE HTTPS implementation, because OVMF already contains the edk2 HTTPS BOOT feature (assuming you build OVMF with "-D NETWORK_HTTP_BOOT_ENABLE -D NETWORK_TLS_ENABLE"). The whole point of CONFIG=qemu is to strip the UEFI build of iPXE to a mere Simple Network Protocol driver. Such a build provides only the lowest level hardware abstraction for UEFI, and the rest (IPv[46], TCP/UDP, DHCP, PXE, TFTP, HTTP(S)) are provided on top by the edk2 implementations of various standard UEFI interfaces."
* Never the less, if you seek regressions then in the UEFI handling of
HTTPS boot.
[Other Info]
* This happens with some OVMF versions like the one I have provided for
debugging but not with the OVMF delivered as part of Focal. Therefore we
should fix it but priority isn't that high.
* Quote from the discussion: "If you want to run a full-featured iPXE
build on a UEFI machine (including: in an OVMF guest), you still can,
of course; lots of people do that, for good reasons. But that use case
is best served by the *standalone UEFI application* build of iPXE
(produced at "bin-x86_64-efi/ipxe.efi" by "make"). The UEFI *driver*
build of iPXE should be as minimal as possible, in comparison -- just
provide SNP for the desired NIC models."
=> Only the EFI roms are changed, neither the PCI roms nor the standalon
IPXE builds will be modified.
---
The version of QEMU (4.2.0) packaged for Ubuntu 20.04 hangs indefinitely at boot if an OVMF bios is used. This happens ONLY with qemu-system-x86_64. qemu-system-i386 works fine with the latest ia32 OVMF bios.
NOTE[1]: the same identical OVMF bios works fine on QEMU 2.x packaged with Ubuntu 18.04.
NOTE[2]: reproducing the fatal bug requires *no* operating system:
qemu-system-x86_64 -bios OVMF-pure-efi.fd
On its window QEMU gets stuck at the very first stage:
"Guest has not initialized the display (yet)."
NOTE[3]: QEMU gets stuck no matter if KVM is used or not.
NOTE[4]: By adding the `-d int` option it is possible to observe that QEMU is, apparently, stuck in an endless loop of interrupts. For the first few seconds, registers' values vary quickly, but at some point they reach a final value, while the interrupt counter increments:
2568: v=68 e=0000 i=0 cpl=0 IP=0038:0000000007f1d225 pc=0000000007f1d225 SP=0030:0000000007f0c8d0 env->regs[R_EAX]=0000000000000000
RAX=0000000000000000 RBX=0000000007f0c920 RCX=0000000000000000 RDX=0000000000000001
RSI=0000000006d18798 RDI=0000000000008664 RBP=0000000000000000 RSP=0000000007f0c8d0
R8 =0000000000000001 R9 =0000000000000089 R10=0000000000000000 R11=0000000007f2c987
R12=0000000000000000 R13=0000000000000000 R14=0000000007087901 R15=0000000000000000
RIP=0000000007f1d225 RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
CS =0038 0000000000000000 ffffffff 00af9a00 DPL=0 CS64 [-R-]
SS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
DS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
FS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
GS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 00000000079eea98 00000047
IDT= 000000000758f018 00000fff
CR0=80010033 CR2=0000000000000000 CR3=0000000007c01000 CR4=00000668
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000044 CCD=0000000000000000 CCO=EFLAGS
EFER=0000000000000d00
NOTE[5]: Just to better help the investigation of the bug, I'd like to remark that the issue is NOT caused by an endless loop of triple-faults. I tried with -d cpu_reset and there is NO such loop. No triple fault whatsoever.
NOTE[6]: The OVMF version used for the test has been downloaded from:
https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20200515.1398.g6ff7c838d0.noarch.rpm
but the issue is the same with older OVMF versions as well.
Please take a look at it, as the bug is NOT a corner case. QEMU 4.2.0 cannot boot with an UEFI firmware (OVMF) while virtualizing a x86_64 machine AT ALL.
Thank you very much,
Vladislav K. Valtchev |
[Impact]
* Booting some OVMF through the efi roms in our ipxe-qemu package triggers
a bad ordering of TPL manipulations and eventually gets the boot stuck.
* It was analyzed by Lazlo that this was due to HTTPS being enabled in the
rom itself triggering this fail when gathering entropy. In addition
isn't required to have HTTPS enabled in this part of the roms as the EFI
payload would be able to handle that on it's own (inside OVMF file from
src:edk2 instead of the combined rom like efi-e1000.rom).
* Disabling HTTPS in the efi-*.rom files should have no functional
impact but mitigate the TPL issues we are seeing.
[Test Case]
* I have provided a test OVMF load attached to this bug
https://bugs.launchpad.net/ipxe/+bug/1882671/+attachment/5387066/+files/OVMF-14c7ed8b51f6-DEBUG-enabled.fd
* Start that in qemu like:
$ qemu-system-x86_64 -enable-kvm -monitor stdio -drive if=pflash,snapshot=on,format=raw,file=/root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd -global e1000.romfile=/usr/lib/ipxe/qemu/efi-e1000.rom -debugcon file:debug.log -global isa-debugcon.iobase=0x402
# /usr/lib/ipxe/qemu/efi-e1000.rom is from package ipxe-qemu
# /root/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd is the debug build I
attached
Symptoms when failing:
- UI never leaves the "initializing" state
- in the debug.log the bad case asserts with
ASSERT /root/edk2/MdeModulePkg/Core/Dxe/Image/Image.c(1676):
Image->Tpl == gEfiCurrentTpl
* Extra Test: HTTPS boot a uEFI guest with the efi roms from ipxe-qemu
with old/new ipxe-qmeu code. This shall ensure that OVMF can really take
over as-is or if we need bug 1883114 before we can do so.
Details TBD when I'm doing these tests
* We pad the rom sizes to be sure, but never the less double check
migrations between Bionic <-> Focal (known to break on size changes)
[Regression Potential]
* Messing around with rom options always is a complex topic, but I'd like
to quote Lazlo (who understands all of this better anyway):
"you don't need the iPXE HTTPS implementation, because OVMF already contains the edk2 HTTPS BOOT feature (assuming you build OVMF with "-D NETWORK_HTTP_BOOT_ENABLE -D NETWORK_TLS_ENABLE"). The whole point of CONFIG=qemu is to strip the UEFI build of iPXE to a mere Simple Network Protocol driver. Such a build provides only the lowest level hardware abstraction for UEFI, and the rest (IPv[46], TCP/UDP, DHCP, PXE, TFTP, HTTP(S)) are provided on top by the edk2 implementations of various standard UEFI interfaces."
* Never the less, if you seek regressions then in the UEFI handling of
HTTPS boot.
[Other Info]
* This happens with some OVMF versions like the one I have provided for
debugging but not with the OVMF delivered as part of Focal. Therefore we
should fix it but priority isn't that high.
* Quote from the discussion: "If you want to run a full-featured iPXE
build on a UEFI machine (including: in an OVMF guest), you still can,
of course; lots of people do that, for good reasons. But that use case
is best served by the *standalone UEFI application* build of iPXE
(produced at "bin-x86_64-efi/ipxe.efi" by "make"). The UEFI *driver*
build of iPXE should be as minimal as possible, in comparison -- just
provide SNP for the desired NIC models."
=> Only the EFI roms are changed, neither the PCI roms nor the standalon
IPXE builds will be modified.
* It makes sense to keep this longer in -proposed to increase the
chance of more people testing this. Therefore I'd ask to accept this
early. I can do my (extended) tests against -proposed and we'd get
some community coverage.
---
The version of QEMU (4.2.0) packaged for Ubuntu 20.04 hangs indefinitely at boot if an OVMF bios is used. This happens ONLY with qemu-system-x86_64. qemu-system-i386 works fine with the latest ia32 OVMF bios.
NOTE[1]: the same identical OVMF bios works fine on QEMU 2.x packaged with Ubuntu 18.04.
NOTE[2]: reproducing the fatal bug requires *no* operating system:
qemu-system-x86_64 -bios OVMF-pure-efi.fd
On its window QEMU gets stuck at the very first stage:
"Guest has not initialized the display (yet)."
NOTE[3]: QEMU gets stuck no matter if KVM is used or not.
NOTE[4]: By adding the `-d int` option it is possible to observe that QEMU is, apparently, stuck in an endless loop of interrupts. For the first few seconds, registers' values vary quickly, but at some point they reach a final value, while the interrupt counter increments:
2568: v=68 e=0000 i=0 cpl=0 IP=0038:0000000007f1d225 pc=0000000007f1d225 SP=0030:0000000007f0c8d0 env->regs[R_EAX]=0000000000000000
RAX=0000000000000000 RBX=0000000007f0c920 RCX=0000000000000000 RDX=0000000000000001
RSI=0000000006d18798 RDI=0000000000008664 RBP=0000000000000000 RSP=0000000007f0c8d0
R8 =0000000000000001 R9 =0000000000000089 R10=0000000000000000 R11=0000000007f2c987
R12=0000000000000000 R13=0000000000000000 R14=0000000007087901 R15=0000000000000000
RIP=0000000007f1d225 RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
CS =0038 0000000000000000 ffffffff 00af9a00 DPL=0 CS64 [-R-]
SS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
DS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
FS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
GS =0030 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 00000000079eea98 00000047
IDT= 000000000758f018 00000fff
CR0=80010033 CR2=0000000000000000 CR3=0000000007c01000 CR4=00000668
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000044 CCD=0000000000000000 CCO=EFLAGS
EFER=0000000000000d00
NOTE[5]: Just to better help the investigation of the bug, I'd like to remark that the issue is NOT caused by an endless loop of triple-faults. I tried with -d cpu_reset and there is NO such loop. No triple fault whatsoever.
NOTE[6]: The OVMF version used for the test has been downloaded from:
https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20200515.1398.g6ff7c838d0.noarch.rpm
but the issue is the same with older OVMF versions as well.
Please take a look at it, as the bug is NOT a corner case. QEMU 4.2.0 cannot boot with an UEFI firmware (OVMF) while virtualizing a x86_64 machine AT ALL.
Thank you very much,
Vladislav K. Valtchev |
|
| 2020-07-07 22:07:56 |
Brian Murray |
ipxe (Ubuntu Focal): status |
Triaged |
Fix Committed |
|
| 2020-07-07 22:07:58 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2020-07-07 22:08:00 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2020-07-07 22:08:05 |
Brian Murray |
tags |
block-proposed block-proposed-focal |
block-proposed block-proposed-focal verification-needed verification-needed-focal |
|
| 2020-07-16 12:28:01 |
Christian Ehrhardt |
ipxe (Ubuntu): status |
Fix Released |
In Progress |
|
| 2020-07-16 12:28:04 |
Christian Ehrhardt |
bug task deleted |
ipxe (Ubuntu Eoan) |
|
|
| 2020-07-16 12:28:13 |
Christian Ehrhardt |
ipxe (Ubuntu Focal): status |
Fix Committed |
In Progress |
|
| 2020-07-16 12:28:15 |
Christian Ehrhardt |
ipxe (Ubuntu Focal): importance |
Low |
Medium |
|
| 2020-07-16 12:28:17 |
Christian Ehrhardt |
ipxe (Ubuntu): importance |
High |
Medium |
|
| 2020-07-16 12:39:24 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/ipxe/+git/ipxe/+merge/387521 |
|
| 2020-07-16 15:15:03 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/ipxe/+git/ipxe/+merge/387531 |
|
| 2020-07-17 04:45:16 |
Christian Ehrhardt |
tags |
block-proposed block-proposed-focal verification-needed verification-needed-focal |
verification-failed verification-failed-focal |
|
| 2020-07-17 06:34:39 |
Launchpad Janitor |
ipxe (Ubuntu): status |
In Progress |
Fix Released |
|
| 2020-07-21 22:17:12 |
Brian Murray |
ipxe (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2020-07-21 22:17:26 |
Brian Murray |
tags |
verification-failed verification-failed-focal |
verification-needed verification-needed-focal |
|
| 2020-07-27 05:10:55 |
Christian Ehrhardt |
tags |
verification-needed verification-needed-focal |
verification-done verification-done-focal |
|
| 2020-08-10 13:06:47 |
Launchpad Janitor |
ipxe (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2020-08-10 13:06:49 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2020-08-12 11:41:31 |
Laszlo Ersek (Red Hat) |
ipxe: status |
New |
Fix Committed |
|
